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I  Introduction 
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The  main  characteristic  which  distinguishes  Bayesian 
estimation  from  the  maximum  likelihood  estimation  is  that  in 
Bayesian  estimation  we  use  the  information  given  by  the  prior 
in  one  way  or  another,  in  addition  to  the  information  which  is 
obtained  directly  from  the  set  of  observations.  It  appears  to  be  a 
common  belief  among  many  researchers  who  are  engaged  in  ability 
measurement  that  Bayesian  estimation  is  superior  to  the  maximum 
likelihood  estimation,  by  virtue  of  this  additional  resource  of 
information,  the  prior.  In  the  area  of  computerized  adaptive 
testing,  for  example,  many  researchers  have  used,  and  are  using, 
Owen's  method  of  Bayesian  estimation  (Owen,  1975),  in  order  to 
accurately  estimate  the  ability  level  of  an  individual. 

It  appears  only  logical  to  correct  this  common  belief, 
however,  and  to  say  that  the  additional  resource  of  information 
is  valuable  and  desirable,  only  if  it  provides  us  with  a_  right 
kind  of  information .  If  this  is  not  the  case,  the  additional 
resource  of  information  is  nothing  but  an  obstacle,  which  may 
contaminate  the  estimation  and  lead  us  to  biases,  inefficiencies, 
and  many  other  undesirable  characteristics.  We  must  pay  our 
attention  to  this  possibility,  since  not  only  such  a  resource 
of  information  will  create  contradictions  in  theory,  but  it  may 
lead  to  serious  social  issues,  such  as  the  unfairness  in  personnel 
selection,  etc.  The  objectivity  of  testing  can  be  phrased  in  the 
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principle  of  treating  all  the  Individuals  of  the  same  level  of 
ability  fairly  and  equally.  If  some  statistical  theory  fails 
in  fulfilling  the  requirement  of  this  principle,  then  the 
resultant  social  issues  are  originated  in  the  theory  itself. 

In  the  present  paper,  the  effect  of  priors  in  Bayesian 
estimation  will  be  considered,  mainly  from  the  standpoint  of 
objective  testing,  which  is  closely  related  with  the  unbiasedness 


of  estimation. 


II  Bayesian  Estimators  Vs .  Maximum  Likelihood  Estimator 

The  main  characteristic  which  distinguishes  Bayesian 
estimation  from  the  maximum  likelihood  estimation  is  that  the 
former  uses  the  prior  as  a  part  of  the  observation  upon  which 
the  estimation  is  made,  whereas  the  latter  does  not.  In  the 
estimation  of  a  parameter  which  belongs  to  an  individual,  this 
prior  is,  in  most  cases,  the  density  function  of  the  population 
to  which  the  individual  belongs.  We  can  say,  therefore,  that 
the  maximum  likelihood  estimation  is  a  population-free  estimation, 
while  Bayesian  estimation  is  not . 

In  estimating  the  examinee's  ability  0  ,  the  maximum 
likelihood  estimator  is  the  point  of  0  which  maximizes  the 
likelihood  function,  L(0)  .  When  the  estimation  is  based  upon 
the  response  pattern,  V  ,  by  virtue  of  the  local  independence 
(Lord  and  Novick,  1968),  we  can  write  for  the  likelihood  function 
L(0) 

(2.1)  L(0)  =  Pv(0)  -  H  Pv  (0)  , 


where  P  (0)  is  the  operating  characteristic  of  the  item  response 

Xg 

x  to  item  g  ,  or  the  conditional  probability  with  which  the 

S 

examinee  obtains  the  item  score  x^  ,  given  ability  0  ,  and 
P„(0)  is  the  operating  characteristic  of  the  response  pattern  V  . 


It  is  well-known  that  the  maximum  likelihood  estimator 


is  asymptotically  unbiased  and  normally  distributed,  when 
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observations  are  taken  from  identical  distributions  (e.g.,  Kendall  & 

Stuart,  1961).  This  implies  that,  when  the  test  consists  of  n 

equivalent  terms,  or  test  items  each  of  which  has  an  identical  set 

of  operating  characteristics,  P  (0)  ,  of  the  items  scores, 

Xg 

x  (=0,1,..., m  )  ,  the  maximum  likelihood  estimate  is  expected 
S  S 

to  be,  approximately,  equal  to  ability  6  itself,  if  the  number 
of  items,  n  ,  is  large  enough  and  the  amount  of  test  information, 
1(6)  ,  is  substantially  large.  This  characteristic  of  the 
maximum  likelihood  estimate  also  exists  in  a  more  general  situation 
where  the  test  items  are  not  equivalent  (cf.  Samejima,  1975). 

It  has  been  shown  (Samejima,  1975,  1977a,  1977b)  that  this 
property  of  asymptotic  unbiasedness  and  normality  of  the  maximum 
likelihood  estimate  provides  us  with  a  good  approximation  even 
when  the  number  of  test  items  is  relatively  small,  and  is  a  useful 
characteristic  in  developing  methods  for  estimating  the  operating 
characteristics  of  graded  item  responses  (Samejima,  1977c,  1977d, 
1978a,  1978b,  1978c,  1978d,  1978e,  1978f). 


where  f(9)  is  the  density  function  of  6  ,  or  the  prior.  This 
is  the  estimator  which  makes  the  expectation  of  the  mean  square 
error,  such  that 
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GO 

(2.3)  E[0*  -  612  f (8)  d6  , 

—GO 

✓ 

where  8*  is  any  estimator  of  0  based  upon  the  response  pattern 
V  ,  minimal  (cf.  Samejima,  1969). 

Bayes  modal  estimator,  9^  ,  of  ability  0  is  the  point 
of  0  at  which  the  function  B^(8)  ,  which  is  defined  by 

(2.4)  Bv(0)  =  Pv(0)  f (0)  , 

is  maximal.  This  estimator  is  similar  to  the  maximum  likelihood 
estimator  in  the  sense  that  it  maximizes  a  "likelihood"  of  a 
given  response  pattern  V  .  Unlike  the  maximum  likelihood  estimator, 
however,  Bayes  modal  estimator,  as  well  as  Bayes  estimator, 
accompanies  a  certain  bias  which  is  caused  by  the  prior,  and 
the  speed  of  convergence  of  the  conditional  distribution,  given 
0  ,  to  the  unbiased  normality  is  slower,  the  characteristic  which 
will  be  observed  and  discussed  in  the  following  chapters. 

Comparison  of  these  three  estimators  reveals  that  Bayes 
estimator,  y^v(8)  ,  assumes  a  unique  finite  value  under  the 
most  general  condition.  A  sufficient  condition  under  which  a 
unique  maximum  likelihood  estimate  is  assured  for  every  possible 
response  pattern  has  been  pursued  (Samejima,  1969,  1972,  1973a, 

1973b,  1974),  and  it  has  been  pointed  out  that  some  widely  used 
models  like  the  normal  ogive  model  and  the  logistic  model 
satisfy  this  condition,  while  the  same  is  not  true  with  the  three- 
parameter  normal  ogive  and  logistic  models  (Bimbaum,  1968). 
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It  is  noted,  however,  that  in  models  like  the  normal  ogive  and 
logistic  models  the  maximum  likelihood  estimate  is  negative 
infinity  for  the  response  pattern  which  consists  of  n  zeros, 
and  that  for  the  response  pattern  whose  elements  are  the  n 
highest  item  scores,  m^  (g=l,2, . . . ,n)  ,  is  positive  infinity. 

A  sufficient  condition  under  which  a  unique  Bayes  modal  estimate 
exists  for  every  possible  response  pattern  has  also  been 
investigated.  It  has  been  pointed  out  that,  if,  in  addition  to 
the  sufficient  condition  for  the  unique  maximum  likelihood 
estimate,  the  first  derivative  of  log  f (6)  is  strictly 
decreasing  in  9  ,  a  unique  Bayes  modal  estimate  exists  for  every 
possible  response  pattern.  Unlike  the  maximum  likelihood  estimate, 
Bayes  modal  estimate  is  finite  even  for  the  above  two  extreme 
response  patterns. 
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III  Objective  Testing  and  Bayesian  Estimation  of  Ability 

We  can  say  that  the  purpose  of  objective  testing  is  to 
measure  an  individual's  ability  without  biases  of  any  kinds. 

It  is  a  common  tendency  that  the  graduate  schools  of  many 
universities  of  the  United  States  adopt  the  Graduate  Record 
Examinations  given  by  Educational  Testing  Service  as  one  of  the 
criteria  in  their  decision  of  accepting  or  rejecting  applicants, 
in  preference  to  similar  tests  developed  and  used  within  each 
college.  This  fact  can  be  considered  as  an  example  in  which 
effort  is  taken  to  avoid  possible  biases  caused  by  different 
tests  and/or  different  norm  groups,  in  order  to  measure  the 
individual's  ability  objectively.  It  is  well-known  that  some 
tests  are  culturally  biased,  and  the  use  of  such  tests  will 
result  in  overestimating  the  ability  levels  of  individuals  with 
some  particular  cultural  backgrounds,  and  in  underestimating 
those  of  individuals  with  some  other  cultural  backgrounds.  This 
second  example  illustrates  a  bias  which  is  rooted  in  the  contents 
of  tests. 

There  is  a  completely  different  type  of  bias,  which  tends 
to  be  overlooked  by  psychologists  and  other  researchers,  but  which 
affects  the  ability  measurement  just  as  strongly.  Suppose  that 
the  content  of  our  test  is  perfectly  valid  and  unbiased.  Using 
such  a  test,  however,  we  may  still  result  in  performing  a  biased 
measurement,  which  is  far  from  the  purpose  of  objective  testing, 
provided  that  we  fail  to  choose  a  right  method  of  estimating 

J 
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the  examinees'  ability  levels.  Thus  the  third  type  of  bias  is  not 
related  with  the  content  of  the  test,  but  with  the  theory  behind 
the  method  of  estimating  the  examinees'  ability,  which  we  adopt 
in  the  process  of  analyzing  our  data. 

We  note  that  the  maximum  likelihood  estimation  does  not, 
in  its  basis,  include  any  information  from  the  population  to 
which  the  individual  belongs,  and,  most  importantly,  there  is  no 
possibility  that  the  resulting  estimate  is  influenced  by  anything 
other  than  the  examinee' s  performance  itself.  The  same  is  not  true 
with  Bayesian  estimation,  however. 

It  has  been  shown  (Samejima,  1969)  that,  using  LIS-U 
(Indow  and  Samejima,  1962,  1966)  and  other  short  tests  as  examples, 
both  the  regression  of  the  Bayes  estimate  and  that  of  the  Bayes 
modal  estimate,  on  ability  0  ,  which  are  given  by 

(3.1)  E(u^v(0) |0)  =  Z  V'v(0)  Pv(0) 

and 

(3.2)  E(9 J9)  =10  P  (0)  , 

V 

respectively,  regress  toward  y  ,  when  the  prior  is  the  normal 
density,  n(y,a)  .  Since  these  two  sets  of  results  are  similar  to 

each  other,  in  this  chapter,  we  shall  use  only  one  estimator, 
i.e. ,  Bayes  modal  estimator,  to  observe  the  biases. 
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Table  3-1  presents  the  discrimination  parameter,  ag  , 
and  the  difficulty  parameter,  bg  ,  of  each  of  the  seven  binary 
test  items  of  LIS-U  ,  which  follows  the  normal  ogive  model  such 
that 


(3.3) 


P  (Q)  = 
g  S2TT 


ve-v  e-»V2  du 


where  P  (0)  is  the  operating  characteristic  for  x  =1  of 
g  g 

binary  item  g  ,  or  the  item  characteristic  function.  The 


information  function,  Ig(6)  »  °f  item  g  is  defined  by 


the 

item 


(3.4) 


I  (0) 

g 


l  I  (0)  P  (0)  =  1(0) [1-P  (0) ] + I  (0)P  (6)  , 
xg=0  xg  xg  0  8  1  8 


where  1^  (0)  is  the  item  response  information  function,  which 
g 

is  given  by 


(3.5)  I  (0)  =  — ~r  log  P  (6) 

e  362  xc 


M3"  log  [1-Pg(9>]  Xg  -  0 


w 108  ye)  *8  ■ 1 


The  test  information  function,  1(0)  ,  can  be  written  as  the  sum 
of  the  item  information  functions,  such  that 


1(0)  =  l  I  (0)  . 

g=l  8 


(3.6) 


TABLE  3-1 


Item  Discrimination  Parameter,  a  ,  and 

g 

Item  Difficulty  Parameter,  ,  of  Each 
of  the  Seven  Items  of  LIS-U  . 


Item 

3g 

bs 

i 

-0.860 

2 

/ 

-0.520 

3 

1.020 

-0.220 

4 

0.800 

-0.030 

5 

1.111 

0.190 

6 

1.389 

0.470 

7 

1.370 

0.760 

! 


Figure  3-1  presents  the  test  information  function  of  LIS-U  ,  and 
its  square  root,  which  is  considered  as  the  reciprocal  of  the 
standard  error  of  estimation  defined  as  a  function  of  ability  0  . 

The  regression  of  the  Bayes  modal  estimator,  *'  ,  on 

ability  3  ,  which  is  given  in  (3.2),  was  obtained  by  using  each 
of  the  four  different  priors,  n(0. 0,1.0)  ,  n(-l. 0,1.0)  , 

n(l. 0,1.0)  and  n(0.0,0.5)  .  These  four  regressions  are  shown 
in  Figure  3-2.  For  convenience,  hereafter,  we  shall  call  these 
four  cases  Cases  1,  2,  3  and  4  . 

We  can  see  in  Figure  3-2  that  these  four  conditional 
means  of  0  are  substantially  different  from  one  another,  in 
spite  of  the  fact  that  they  are  all  estimates  of  0  obtained 
through  the  same  test,  LIS-U  .  We  note,  in  addition,  that 

none  of  these  four  regressions  is  close  to  the  straight  line, 
which  is  drawn  by  a  solid  line  In  Figure  3-2  indicating  the 
unbiasedness  of  estimation,  and  the  discrepancies  are  large 
for  values  of  6  which  are  far  from  the  mean  of  each  prior. 
Discrepancies  among  the  four  conditional  means  are  great  even 
at  8  ■  0  ,  where  the  test  information  function,  1(0)  ,  of 
LIS-U  assumes  as  high  a  value  as  5.55546  ;  the  fact  which 
indicates  strong  biases  of  estimation,  i.e.,  the  expectation 
of  ability  is  0.00299,  -0.16252,  0.16915  or  0.00126  , 
depending  upon  the  prior  to  which  examinees  of  ability  0.0 
are  assigned  to.  Thus  the  examinees  who  belong  to  Case  2  are 
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FIGURE  3-1 


Test  Information  Function  (Solid  Line)  and  Its 
Square  Root  (Dotted  Line)  of  LIS-U  . 
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severely  and  unqualifiedly  handicapped  while  those  who  belong  to 
Case  3  are  allowed  to  enjoy  the  advantage,  regardless  of  the  fact 
that  they  are  individuals  whose  ability  levels  are  exactly  the  same. 

These  discrepancies  in  regression  are  enhanced  if  we  shift 
the  true  ability  level  from  0.0  to  0.5  ,  at  which  the  test 
information  function  assumes  4.93877  ,  slightly  less  than 
5.55546  at  8  *  0.0  .  The  expected  values  of  the  Bayes  modal 
estimate  are  0.44410,  0.26827,  0.63169  and  0.28436  for  Cases 
1,  2,  3  and  4,  respectively,  providing  us  with  the  range  of 
0.36342  .  Greater  discrepancies  are  observed,  however,  at 
levels  of  9  where  the  amount  of  test  information  is  much 
smaller.  At  9  *  3.0  where  1(8)  **  0.08634  ,  for  example,  the 
expected  Bayes  modal  estimates  are  1.34995,  1.07489,  1.70326 
and  0.77629  ,  respectively,  none  of  which  is  close  to  the  true 
ability  level,  3.0  ;  at  8  =  -2.0  where  1(6)  *  0.96397  ,  the 
expected  estimates  are  -1.26590,-1.61087,  -1.00075  and 
-0.73634  .  We  can  see  that  at  these  ability  levels  Bayes  modal 
estimate  is  at  the  mercy  of  a  given  prior;  the  fact  which  is 
against  the  principle  of  objective  testing. 

We  note  that  the  expected  Bayes  modal  estimates  are  0.00299 
and  0.00126  at  0  ■  0.0  for  Cases  1  and  4,  respectively,  both 
of  which  are  very  close  to  the  true  ability  level,  0.0  ,  whereas 
in  Cases  2  and  3  the  expected  estimates  are  -0.16252  and  0.16915  , 
which  are  farther  from  0.0  in  the  two  directions.  These  biases 
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for  Cases  1  and  4  come  from  the  priors,  i.e. ,  n(0. 0,1.0)  and 

n(0.0,0.5)  respectively,  showing  regressions  toward  the 
means  of  the  separate  priors.  Similar  tendencies  are 
observed  at  0  *  -1.0  and  0  =  1.0  ,  where  the  means  of  the 
priors  for  Cases  2  and  3  are  located,  respectively;  the 
expected  Bayes  modal  estimates  are  -0.85033,  -1.09301, 

-0.64375  and  -0.52486  at  0  =  -1.0  ,  and  0.84548,  0.63805, 
1.08443  and  0.51965  at  0  =  1.0  ,  for  Cases  1,  2,  3,  and  4. 

It  is  evident  from  the  above  observations  that,  even  if 
the  test  itself  is  perfectly  objective  in  content,  the  use  of 
Bayes  modal  estimator  of  ability  will  destroy  the  objectivity  of 
testing,  providing  the  examinees  with  unqualified  advantages  or 
disadvantages,  depending  upon  the  relative  positions  of  their 
ability  levels  and  the  prior  to  which  they  are  assigned. 

As  was  mentioned  earlier,  the  maximum  likelihood  estimator 
is  asymptotically  unbiased,  the  characteristic  which  suits  the 
principle  of  objective  testing,  although  for  short  tests  the 
approximation  may  not  be  very  good.  It  will  be  worthwhile, 
therefore,  to  investigate  the  destruction  of  objectivity  by 
the  Bayes  modal  estimator  in  comparison  with  the  behavior  of 
the  maximum  likelihood  estimator. 

Figure  3-3  presents  four  functions,  i.e.,  the  standard 
normal  density  function,  n(0,l)  (solid  line),  and  three 
approximations  to  n(0,l)  .  Each  of  these  three  approximations 
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Is  the  product  of  two  functions,  P^(6)  an<^  [l~Pj(9)l  »  which 
are  given  by  the  normal  ogive  functions  such  that 


3h(6-V  , 

«•’>  V>  -  -k  I  w  * 


and 


(3.8) 


Pj(0) 


1 

/iTT 


(  a  (6-b  ) 

2  2  -u2/2  j 

e  du  , 


where  a^  -  a^  and  b^  *»  -b^  .  These  two  parameters,  and  b^  , 
are  0.94810  and  -0.35454  for  the  function  drawn  by  a  dotted  line 
in  Figure  3-3,  0.94980  and  -0.35391  for  the  one  drawn  by  a 
broken  or  long,  dashed  line,  and  0.95259  and  -0.35287  for  the 
one  drawn  by  a  short,  dashed  line,  respectively.  These  three 
approximations  are  obtained  by  setting  the  product  of  the  two 
functions  equal  to  the  standard  normal  density  function  at 
0  *  0.3  ,  0  ■  0.6  and  9  »  0.9  ,  respectively,  in  addition 
to  0  •  0.0.  We  notice  that  these  four  curves,  including  n(0,l)  , 
in  Figure  3-3  are  practically  indistinguishable. 

We  notice  that  the  formulas  in  (3.7)  and  (3.8)  are  identical 
with  the  item  characteristic  function  in  the  normal  ogive  model 
on  the  dichotomous  response  level,  which  is  shown  as  (3.3). 

This  implies  that  the  prior,  n(0,l)  ,  is  practically  the  same 
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as  the  product  of  the  two  operating  characteristics  of  the 
hypothetical  binary  items,  h  and  j  ,  for  the  response  pattern, 
(1,0)  .  The  Bayes  modal  estimator  with  the  prior  n(0,l)  can 
be  considered,  therefore,  as  the  maximum  likelihood  estimator, 
obtained  from  the  response  pattern  on  LIS-U  plus  additional 
two  responses,  1  and  0  ,  to  the  hypothetical  binary  items, 
h  *nd  j  .  Note  that  chese  two  additional  item  responses  are 
always  1  and  0  ,  regardless  of  the  true  ability  level. 

Let  V*  denote  any  response  pattern  on  the  two  hypothetical 
test  items,  h  and  j  .  Since  both  are  binary  items,  there 
are  only  four  possible  response  patterns  V*  ,  i.e.,  (0,0), 

(0,1),  (1,0)  and  (1,1)  .  The  operating  characteristic, 

Pv*(0)  ,  of  the  response  pattern  V*  is  given  by 


(3.9) 


v(0)  < 


d-phce)  ]  ti-Pj  O)  ] 

for 

V*'=(0,0) 

[1-Ph(0)j  p^e) 

for 

r—4 

o 

II 

* 

> 

ph(9)  [i-p^oh 

for 

V*'«(1,0) 

?h(9)  Pj (0) 

for 

v*'»(l,l) 

where  P, (9)  and  P,(9)  are  the  item  characteristic  functions 

h  J 

of  the  hypothetical  binary  items,  h  and  j  ,  which  are  given 
by  (3.7)  and  (3.8),  respectively.  Figure  3-4  presents  the 
operating  characteristics  of  the  four  response  patterns  with 
a,  -  a.  -  0.95  and  b,  -  -b.  -  -0.35  . 


PROBABILITY 
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FIGURE  3-4 

Operating  Characteristics,  Pv*(0)  ,  of  Four  Possible  Response 

Patterns,  (0,0)  ,  (0,1)  ,  (1,0)  and  (1.1)  ,  of  Hypothetical 
Biziary  Items,  h  and  j  . 
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As  we  can  see  in  Figure  3-4,  although  the  probability  is 
highest  at  0  =  0.0  for  V*'=  (1,0)  compared  with  those  for 
the  other  three,  i.e.,  0.397  against  0.233  for  V*'  =  (0,0)  , 
0.137  for  V*’=  (0,1)  and  0.233  for  V*'=  (1,1)  ,  and  this 
tendency  holds  in  the  vicinity  of  this  ability  level,  at 
9  -  -0.4  and  0  =  0.4  it  is  already  exceeded  by  the 
probabilities  for  V*'  =  (0,0)  and  V*'  =  (1,1)  ,  respectively, 
i.e.,  0.367  against  0.395  in  each  case.  If  we  shift  the 

ability  level  from  0.0  to  ±1.0  ,  ±1.5  ,  ±2.0  ,  ±2.5  ,  ±3.0 
and  ±3.5  ,  the  probability  for  V*'=  (1,0)  decreases  rapidly 
relative  to  either  the  probability  for  V*’=  (1,1)  or  the  one 
for  V*'=  (0,0)  ,  i.e.,  0.242  vs.  0.659  ,  0.132  vs.  0.829  , 

0.058  vs.  0.929  ,  0.020  vs.  0.976  ,  0.006  vs.  0.993  and 
0.001  vs.  0.998  ,  respectively.  In  other  words,  at  0  =  2.0  , 
for  example,  chances  are  only  58  times  out  of  1,000  that  the 
examinee  of  this  ability  level  obtains  (1,0)  for  V*'  ,  in 
comparison  with  929  times  out  of  1,000  for  V*'**  (1,1)  . 

As  far  as  we  use  Bayes  modal  estimator,  however,  it  is  treated 
as  if  chances  were  1,000  times  out  of  1,000  for  the  examinee 
of  this  ability  level  to  obtain  (1,0)  for  V*'  !  It  is  no 
wonder  that  the  conditional  expectation  of  Lhe  Bayes  modal 
estimate,  given  8  ,  regresses  toward  the  center,  which  indicates 
the  tendency  that  examinees  of  lower  ability  levels  obtain  higher 
values  of  the  Bayes  modal  estimate  and  those  of  higher  levels 


obtain  lower  values  of  the  Bayes  modal  estimate.  The  effect 
must  be  especially  strong  when  the  number  of  items  in  the  test 
is  relatively  small.  Note  that  this  tendency  is  relative  to 
the  prior,  to  which  the  individual  is  assigned.  In  other  words, 
the  hypothetical  test  items  h  and  j  ,  whose  item  characteristic 
functions  approximate  the  prior,  differ  from  one  population  to 
another,  the  fact  which  explains  the  relative  positions  of  the 
four  regressions  in  Figure  3-2  .  This  additional  response 
pattern,  (1,0)  ,  for  the  hypothetical  test  items  h  and  j 
creates  nothing  but  biases  which  contradict  the  principle  of 
objective  testing. 

Although,  in  the  above  examples,  normal  density  functions 
were  solely  used  for  priors,  we  can  see  that  the  same  logic 
can  be  applied  for  priors  of  different  shapes.  The  resultant 
bias  caused  by  Bayesian  estimation  depends  upon  the  particular 
shape  of  the  prior,  and  the  set  of  hypothetical  items  and  the 
specific  response  pattern,  whose  operating  characteristic 
approximates  the  prior. 
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IV  Bayes  Modal  Estimate  When  the  Amount  of  Test  Information  Is 
Large 

In  the  preceding  chapter,  we  observed  the  bias  caused  by  the 
Bayesian  estimation  using  Bayes  modal  estimator  with  a  relatively 
short  test,  LIS-U,  Since  most  priors  can  be  approximated  by  the 
product  of  the  operating  characteristics  of  a  relatively  small 
number  of  hypothetical  items,  it  is  expected  that  the  effect  of  a 
prior  will  be  less  dominating  in  the  resultant  estimation  of  ability 
if  the  test  is  longer  and  more  informative,  i.e.,  if  the  test 
information  function  assumes  high  values  for  the  entire  range  of 
ability  of  our  interest.  In  this  chapter,  therefore,  we  shall 
observe  the  effect  of  priors  in  Bayesian  estimation  using  two 
hypothetical  tests.  Test  A  and  Test  B,  each  of  which  provides  us 
with  an  approximately  constant  amount  of  test  information,  21.6  , 
for  the  interval  of  0  ,  [-3,0,  3.0]  (cf.  Samejima,  1977c).  Test 
A  consists  of  thirty-five  graded  test  items  with  mg=2  for  each 
item,  while  Test  B  consists  of  twenty  items  with  m^=3  for  every 
item.  All  of  these  graded  test  items  follow  the  normal  ogive 
model  on  the  graded  response  level,  whose  operating  characteristic, 


P  (.8)  ,  of  the  item  score  x  (=0,1,..., m  )  is  given  by 
g  8  8 


(4.1) 


px  (9)  = 


V0“V  , 

g  e  du  , 

a  (8-b  .) 

g  x  +1' 


where 


(4.2) 


—co  a  bp  <  < 


<  b  <  b  , . 
m  m  +1 
g  g 


! 
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and 

(4.3)  a  >  0  . 

g 


Table  4-1  presents  the  discrimination  parameter, 

the  two  difficulty  parameters,  b  for  x  =  1,  2  ,  of 

Xg  S 


a  ,  and 
g 

each  of 


the  thirty-five  items  of  Test  A.  Table  4-2  also  presents  the 

discrimination  parameter,  a^  ,  and  the  three  difficulty  parameters, 

b  for  x  =  1,  2,  3  ,  of  each  of  the  twenty  items  of  Test  B. 
xg  g 


For  each  of  the  one  hundred  hypothetical  examinees,  whose 
ability  0  distributes  approximately  normally  (cf.  Samejima, 

1977c)  ,  both  the  maximum  likelihood  estimate  and  Bayes  modal 
estimate  were  obtained  upon  a  response  pattern,  which  was  calibrated 
by  the  Monte  Carlo  method,  for  each  of  Tests  A  and  B.  Figure  4-1 
presents  these  two  estimates  plotted  against  the  true  ability  0 
for  each  of  the  one  hundred  hypothetical  examinees.  We  can  see  in 
these  two  graphs  of  Figure  4-1  that  Bayes  modal  estimate,  which  is 
represented  by  solid  triangles,  tends  to  regress  toward  the  center, 
in  comparison  with  the  maximum  likelihood  estimate,  which  is  drawn 
by  crosses,  for  both  Tests  A  and  B,  although  the  tendency  is  less 
conspicuous  than  in  the  case  of  LIS-U.  THh  sample  linear  regression 
of  any  estimator  6*  on  ability  8  is  given  by  a0  +  a^6  ,  where 


(4.4)  aQ  =  M0*  -  (s0*/s0)  Corr . (9, 6*)  M0 


and 


latent  trait  0 


FIGURE  4-1 


Maximum  likelihood  Estimates  (Crosses)  and  Bayes  Modal  Estimates  (Triangles) 
of  Ability  for  One  Hundred  Hypothetical  Examinees  Whose  Ability  Distributes 
Approximately  H(0,1)  ,  Obtained  Through  Teat  A  . 
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LATENT  TRAIT  0 


FIGURE  4-1  (Continued):  Those  Obtained  through  Test  B  • 
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(A. 5)  -  (sg^/sg)  Corr.(9,9*)  , 

with  M  and  s  representing  the  sample  mean  and  the  sample 
standard  deviation,  respectively.  Replacing  6*  by  the  maximum 

A 

likelihood  estimate  9  ,  the  two  coefficients,  Oq  and  >  of 
the  sample  linear  regression  were  calculated  for  each  of  Tests  A 
and  B,  and  are  presented  in  Table  4-3.  We  can  see  from  this  result 
that  the  sample  linear  regression  for  Test  A  is  almost  identical 
with  the  straight  line  drawn  in  Figure  4-1,  indicating  the  unbiased 
estimation,  since  0.9933  is  very  close  to  unity  and  -0.0088  to 
zero.  For  Test  B,  the  sample  linear  regression  is  flatter  than  the 
line  of  forty-five  degrees,  showing  regression  toward  the  center, 
although  the  degree  of  regression  is  very  small.  The  corresponding 
set  of  coefficients  were  calculated  for  the  Bayes  modal  estimate, 

A 

9  ,  for  both  Tests  A  and  B,  and  are  shown  in  the  same  table.  We 
can  see  that  for  both  tests  Bayes  modal  estimate  tends  to  regress 
toward  the  center  more  strongly.  This  tendency  is  far  less  than  it 
is  for  a  short  test  like  LIS-U,  however,  the  fact  which  is 
anticipated  from  the  difference  in  the  amounts  of  test  information 
for  this  range  of  9  , 

As  another  example,  Bayes  modal  estimate  was  obtained  for 
each  of  five  hundred  hypothetical  examinees,  whose  ability  levels 
differ  from  -2.475  to  2.475  with  the  step  of  0.05  ,  with  five 
examinees  sharing  each  ability  level.  The  estimation  was  made 
upon  the  response  pattern  of  Test  A,  which  was  calibrated  by  the 
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Monte  Carlo  method,  using  each  of  the  four  priors,  n(0. 0,1.0)  , 
n(-l. 0,1.0)  ,  n(l. 0,1.0)  and  n(0.0,0.S)  . 

Figure  4-2  presents  the  mean  of  the  five  Bayes  modal 
estimates  thus  obtained  for  each  of  the  one  hundred  levels  of  6  , 
which  is  plotted  by  a  dot,  together  with  the  corresponding  mean 
of  the  five  maximum  likelihood  estimates,  which  is  represented  by 
a  cross,  for  each  of  the  four  priors.  Examination  of  each  of 
these  four  graphs  reveals  the  anticipated  bias  of  the  Bayesian 
estimation,  i.e.,  the  tendency  to  regress  toward  0.0  ,  -1.0  ,  1.0 
and  0.0  ,  respectively,  although  it  is  even  less  conspicuous  than 
in  the  preceding  example,  except  for  one  case  in  which  the  prior 
is  n(0.0,0.5)  .  The  five  sets  of  coefficients  of  the  sample 
linear  regressions  of  the  five  hundred  estimates  on  ability  were 
calculated  in  the  same  manner  as  in  the  preceding  example,  and  are 
shown  in  Table  4-4.  Again  the  sample  linear  regression  of  the 
maximum  likelihood  estimate  is  practically  identical  with  the 
straight  line  of  forty-five  degrees,  with  the  two  coefficients, 
c*q  *  -0.0058  and  ot^  *  1.0047  ,  being  so  close  to  zero  and  unity, 
respectively,  while  the  other  sets  of  coefficients  for  the  four 
sets  of  Bayes  modal  estimates  indicate  flatter  lines,  suggesting 
separate  and  anticipated  regressions. 

From  these  results,  it  is  obvious  that  even  with  an 
informative  test  having  a  large  amount  of  test  information,  like 
21.6,  for  the  entire  range  of  ability  0  of  our  interest  the 
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FIGURE  4-2  (Continued):  The  Prior  Is  n(l,l)  . 
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FIGURE  4-2  (Continued):  The  Prior  i*  n(0.0,0.5)  . 
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TABLE  4-4 

Coefficients  of  the  Linear  Regression  of  Each 
of  the  Five  Estimators  on  Ability  0  ,  for 
Five  Hundred  Observations  on  Test  A.  The 
Five  Estimators  are  the  Maximum  Likelihood 
Estimator  (MLE)  and  the  Four  Bayes  Modal 
Estimators  with  Different  Priors. 


“o 

ai 

MLE 

-0.0058 

1.0047 

BME 

n(  0.0, 1-0) 

-0.0053 

0.9591 

n(-l. 0,1.0) 

-0.0500 

0.9592 

n(  1.0, 1.0) 

0.0395 

0.9590 

n (  0.0, 0.5) 

-0.0039 

0.8456 
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effect  of  a  prior  in  Bayes  modal  estimation  appears  in  the  fora 
of  bias,  which  was  observed  with  a  shorter  test  like  LIS-U 
whose  maximum  amount  of  test  information  is  5.55546  .  We  can 
see  that,  as  we  increase  the  amount  of  test  information,  Bayes 
modal  estimate  approaches  the  maximum  likelihood  estimate.  This 
implies  that  Bayes  modal  estimate,  too,  has  the  asymptotic 
unbiasedness,  as  the  maximum  likelihood  estimate  does.  The 
convergence  to  the  unbiasedness  is  slower  for  the  Bayes  modal 
estimate,  however,  since  Bayes  modal  estimate  must  "shake  off" 
the  effect  of  the  prior  in  the  process  of  approaching  the 
unbiasedness.  We  can  say  that  the  prior  is  nothing  but  an 
obstacle  whose  effect  should  be  gotten  rid  of  in  order  to  approach 
the  unbiasedness  of  estimation,  which  is  essential  for 
objective  testing. 


r 


V  Effect  of  the  Prior  in  Tailored  Testing 


We  shall  observe  here  how  the  prior  affects  the  resultant 

ability  estimation  in  tailored  testing,  where  a  single  item  is 

selected  from  an  item  pool  and  presented  to  an  individual 

examinee,  sequentially.  A  tailored  testing  situation  is  simulated 

with  an  hypothetical  item  pool,  in  which  there  are  nine  binary 

item  groups,  each  of  which  consists  of  a  large  number  of 

equivalent,  binary  test  items  following  the  normal  ogive  model, 

which  is  given  by  (3.3)  and  whose  item  discrimination 

parameter,  a  ,  and  item  difficulty  parameter,  b  ,  for  the 
6  8 

item  group  g  are  shown  in  Table  5-1.  We  assume  eleven 
hypothetical  examinees,  whose  ability  levels  are  -2.25,  -1.75, 
-1.25,  -0.75,  -0.25,  0.00,  0.50,  1.00,  1.50,  2.00  and  2.50, 
respectively.  We  also  assume  four  different  situations,  in 
one  of  which  the  maximum  likelihood  estimation  is  applied  for 
the  ability  estimation,  and  in  the  other  three  Bayes  modal 
estimation  is  used,  with  three  different  priors,  n(0. 0,1.0)  , 
n(0.0,0.8)  and  n(0.0,0.5)  ,  respectively.  In  the  first 
situation  of  maximum  likelihood  estimation,  an  item  from  group 
5  is  always  chosen  as  the  first  item  to  present  to  an  examinee, 
and,  depending  upon  the  examinee's  response  to  this  item,  the 
second  item  is  chosen  either  from  group  1  or  group  9.  That  is 
to  say,  if  the  examinee's  response  to  the  first  item  is  correct, 
then  the  second  item  is  chosen  from  group  9,  i.e.,  the  most 


-38- 


V-2 


TABLE  5-1 

Item  Discrimination  Parameter,  a  ,  and  Item 

g 

Difficulty  Parameter,  bg  ,  of  Each  of  the  Nine 

Groups  of  Binary  Test  Items  Used  as  the  Item 
Pool  in  the  Simulated  Tailored  Testing. 


Item 

Group 

a8 

bg 

1 

1.20 

-2.00 

2 

1.60 

-1.50 

3 

2.00 

-1.00 

4 

1.40 

-0.50 

5 

1.80 

0.00 

6 

1.30 

0.50 

7 

1.70 

1.00 

8 

1.90 

1.50 
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difficult  item  group,  and,  if  it  is  incorrect,  then  the  second 

item  is  chosen  from  group  1,  the  easiest  item  group.  The 

examinee  will  stay  with  the  same  item  group  for  the  following 

items,  until  he  fails  in  answering  an  item  correctly  if  it  is 

group  1,  and  until  he  succeeds  in  answering  an  item  incorrectly 

if  it  is  group  9.  Thereafter,  since  every  current  likelihood 

function  has  a  local  maximum,  an  item  from  the  item  group  whose 

item  information  function,  I  (0)  ,  which  is  defined  by  (3. A), 

S 

is  the  greatest  at  the  value  of  current  maximum  likelihood 
estimate  is  chosen  and  presented  next,  and  this  will  go  on 
until  the  amount  of  test  information  at  the  current  maximum 
likelihood  estimate  reaches  or  exceeds  a  certain  criterion. 

All  the  responses  of  the  hypothetical  examinees  are  calibrated 
by  the  Monte  Carlo  method. 

In  Bayesian  estimation,  the  first  estimate  is  the  modal 

point  of  the  prior.  The  second  item  is  an  item  chosen  from 

the  item  group  whose  item  information  function,  I  (0)  ,  is 

8 

the  greatest  at  the  modal  point  of  the  prior,  and  the  third 
item  is  from  the  item  group  whose  item  information  function 
is  the  greatest  at  the  current  Bayes  modal  estimate,  and  so 
forth,  and  the  presentation  of  a  new  item  is  terminated  when 
the  amount  of  test  information  at  the  current  estimate  of  the 
examinee's  ability  has  reached  the  same  criterion  used  in  the 


maximum  likelihood  estimation. 
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Figures  5-1  through  5-3  present  the  result  of  these  four 
simulated  tailored  testing  for  each  of  the  eleven  hypothetical 
examinees.  In  each  of  these  three  figures,  the  sequential 
result  of  the  maximum  likelihood  estimation  is  presented  by 
solid  triangles,  and  that  of  one  of  the  three  Bayesian 
estimations  is  shown  by  hollow  circles.  In  each  Bayesian 
estimation,  the  first  circle  is  located  at  the  modal  point  of 
the  prior,  so  the  actual  number  of  test  items  used  in  the 
simulated  tailored  testing  is  one  less  than  the  number  of 
circles.  The  number  of  test  items  which  are  presented  to  each 
examinee  in  each  situation  is  shown  in  parentheses  in  Table  5-2, 
following  the  eventual  ability  estimate.  The  amount  of  test 
information  used  as  the  criterion  for  terminating  the 
presentation  of  a  new  item  in  this  simulated  tailored  testing 
is  20.0. 

As  was  the  case  with  the  previous  examples,  the  effect 
of  a  prior  appears  in  the  form  of  underestimating  the  ability 
levels  of  examinees  which  are  much  higher  than  the  mean  of  the 
prior,  and  of  overestimating  those  which  are  much  lower,  in 
all  three  cases  of  the  Bayesian  estimation,  with  some  exceptions 
at  8  =  -2.25  .  Note  that,  in  comparison  with  these  results, 
errors  of  measurement  in  the  maximum  likelihood  estimation  are 


more  randomly  distributed  in  both  directions.  We  notice, 
moreover,  that  even  in  the  two  exceptions  at  0  =  -2.25.  ,  it 


Tailored  Testing  with  n(0,l)  as  the  Prior,  for 
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FIGURE  5-1  (Continued):  The  Prior  is  n(0,l)  »  9  =  -1.75 
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FIGURE  5-1  (Continued):  The  Prior  Is  n(0,l)  ,  0  =  -0.75 
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FIGURE  5-1  (Continued):  The  Prior  is  n(0,l)  ,  8  =  1.00 
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Successive  Maximum  Likelihood  Estimates  (Triangles)  and  Bayes  Modal  Estimates 
(Circles)  in  the  Simulated  Tailored  Testing  with  n(0.0,0.8)  as  the  Prior  for 
a  Hypothetical  Examinee  Whose  Ability  Level  is  -2.25  . 
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Successive  Maximum  Likelihood  Estimates  (Triangles)  and  Bayes  Modal  Estimates 
(Circles)  in  the  Simulated  Tailored  Testing  with  n(0.0,0.5)  as  the  Prior  for 
a  Hypothetical  Examinee  Whose  Ability  Level  is  -2.25  . 


TABLE  5-2 


I 


took  28  and  35  test  items  to  make  up  for  the  effect  of  the 

priors,  n(0. 0,1.0)  and  n(0.0,0.8)  ,  respectively,  until  the 
estimates,  -2.3161  and  -2.4097  ,  which  are  slightly  less 
than  hut  close  to  the  true  ability  level,  -2.25  ,  were  obtained. 
These  two  numbers  of  items,  28  and  35  ,  are  far  too  large 
compared  with  the  average  number  of  test  items  used  in  these 
forty-four  sequences  which  turned  out  to  be  14.70  . 

It  is  interesting  to  note  that,  when  the  prior  is 
n(0.0,0.5)  ,  it  took  only  16  items  in  the  sequence,  compared 
with  28  and  35  in  the  other  two  Bayesian  estimations, 

before  the  testing  is  completed  for  the  examinee  whose  ability 
level  is  -2.25  .  The  eventual  Bayes  modal  estimate  Is  -1.5683 
however,  which  is  far  away  from  the  true  ability  level  -2.25  ; 
the  fact  which  shows  a  far  stronger  effect  of  the  prior  than 
the  other  two  cases.  This  indicates  that  the  effect  of  the 
prior  was  so  strong  that  the  testing  could  not  "correct"  the 
bias  caused  by  the  prior  throughout  the  whole  process  of 
tailored  testing. 

A  similar  tendency  of  "trade-off"  between  the  accuracy 
of  ability  estimation  and  the  number  of  test  items  exists  in 
the  relationships  among  the  four  ability  estimates  of  the  same 
examinee  when  we  lower  the  criterion  for  terminating  the 
presentation  of  a  new  item.  If  we  use  1(8)  *  15.0  as  the 
criterion  they  are  -2.2352  (18)  ,  -2.2796  (21)  ,  -2.0694  (18) 


I 
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and  -1.4207  (11)  ,  arranged  in  the  order  of  the  maximum 
likelihood  estimate  and  Bayes  modal  estimates  with  n(0. 0,1.0)  , 
n(0.0,0.8)  and  n(0.0,0.5)  as  the  prior,  respectively,  with 
the  corresponding  number  of  items  used  in  each  tailored  testing 
in  parenthesis;  if  we  use  1(6)  =  10.0  they  are  -2.2099  (13)  , 
-1.9812  (12)  ,  -1.7915  (12)  and  -1.1931  (7)  .  The  situation 
will  not  be  improved  even  if  we  switch  the  criterion  to  the 
stability  of  successive  estimates  in  tailored  testing.  If,  for 
example,  we  terminate  the  presentation  of  a  new  item  right 
after  three  successive  estimates  stayed  within  the  range  of 
±0.075  of  the  separate,  preceding  estimates,  the  resulting  four 
estimates  and  the  numbers  of  test  items  for  the  examinee 
whose  ability  level  is  -2.25  are  -2.2994  (11)  ,  -1.9812  (12)  , 
-1.9062  (14)  and  -1.3781  (10)  ,  respectively.  For  the 
examinee  of  the  other  deviated  ability  level,  2.50  ,  using  the 
same  convergence  criterion,  the  results  are  2.8183  (11)  , 

2.1251  (11)  ,  2.1130  (11)  and  1.8152  (15)  ,  compared  with 
those  obtained  with  the  criterion  of  1(0)  =  20.0  ,  which  are 
2.5415  (21)  ,  2.2822  (19)  ,  2.2251  (21)  and  1.9280  (18)  . 

Since  only  3  out  of  1,000  people  are  outside  of  the 
range  of  three  standard  deviations  plus  or  minus  the  mean,  if 
the  population  ability  distribution  is  normal,  chances  are 
very  slim  that  an  examinee  whose  ability  level  is  -2.25  or 
2.50  is  assigned  to  the  prior,  n(0.0,0.5)  , 


or  even 
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n(0.0,0.8)  .  In  practice,  however,  such  a  situation  is  more 
likely  to  happen,  since  the  function  assumed  for  the  prior  is 
more  or  less  arbitrary,  and,  furthermore,  the  assignment  of 
individuals  to  a  specific  prior  itself  is  more  or  less 
arbitrary,  using  their  sex,  ethnic  background,  and  so  forth. 

We  must  say,  therefore,  Bayesian  estimation  applied  for 
personnel  selection,  for  example,  could  cause  a  serious  problem 
of  unfair  discrimination,  even  if  the  content  of  the  test 
itself  is  perfectly  valid. 

The  sequential  results  of  the  four  ability  estimations 
In  the  simulated  tailored  testing  for  the  eleven  hypothetical 
examinees  are  presented  in  Appendix  as  Tables  A-l  through  A-4. 


-78- 


VI-1 


VI  Some  Criticisms  on  the  Common  Belief  in  Bayesian  Estimation 
Many  researchers  who  use  Bayesian  estimates  in  favor  of 
the  maximum  likelihood  estimate  refer  to  the  two  aspects  which 
they  think  are  the  advantages  of  the  Bayesian  estimation  over 
the  maximum  likelihood  estimation.  These  two  aspects  are: 

(1)  While  Bayesian  estimation  provides  us  with  seemingly 

reasonable  finite  values  as  the  estimates  for  all  the 

response  patterns,  the  maximum  likelihood  estimation  gives 

us  positive  and  negative  infinities  for  the  two  extreme 

response  patterns,  (0,0,...,0)'  and  (m  ,m  ,  ...,m  )'  , 

X  Z  n 

respectively. 

(2)  The  frequency  distribution  of  t  le  resultant  set  of 
maximum  likelihood  estimates  is  mure  scattered  than  the 
true  ability  distribution,  or  the  prior,  whereas  that  of 
Bayesian  estimates  is  not. 

It  should  be  recalled  that  with  each  of  the  two 
hypothetical  tests,  i,e..  Tests  A  and  B,  which  were  introduced 
in  Chapter  4,  every  single  hypothetical  examinee  of  the  two 
groups  obtained  a  finite  maximum  likelihood  estimate.  This 
results  from  the  fact  that  the  amount  of  test  information  of 
each  test  lo  as  large  as  21.6  for  the  interval  of  9  ,  (-3.0, 3.0)  , 
the  range  within  which  all  the  examinees'  ability  is  located, 
and  for  this  reason  none  of  the  examinees  obtained  either  of 


i 


* 


I 


the  two  extreme  response  patterns.  In  fact,  it  can  be  proved 
easily  that  the  conditional  probability  with  which  the  examinee 
of  a  given  ability  level  obtains  the  response  pattern,  (0,0, , . . ,0) ' , 
or  (m^jm^, • • . ,n»n) *  ,  approaches  zero  as  the  amount  of  test 
information  increases  at  that  level  of  0  .  It  is  highly  unlikely, 
therefore,  that  an  examinee  whose  ability  level  is  located  in 
the  interval  of  0  ,  (-3.0,3,Q)  ,  will  obtain  one  of  these  two 

extreme  response  patterns,  if  the  test  is  highly  informative 
throughout  the  interval,  as  is  true  with  both  Tests  A  and  B. 

If  the  test  is  short,  like  LIS-U  ,  however,  it  is  more 
likely  that  examinees  obtain  one  of  the  two  extreme  response 
patterns.  This  test,  LIS-U  ,  was  originally  developed  for 
the  purpose  of  classifying  a  group  of  examinees,  whose  ability 
distributes,  approximately,  normally  with  zero  and  unity  as  the 
two  parameters,  into  small  subgroups  of  equal  sizes  (cf.  Indow 
and  Samejiraa,  1962,  1966).  The  test  information  function  assumes, 
therefore,  high  values  around  0  =  0.0  ,  and  lower  values  as  0 
departs  from  zero,  as  we  can  see  in  Figure  3-1,  since  for  the 
above  purpose  it  is  not  important  to  estimate  the  ability  of 
very  bright  or  very  dull  examinees  accurately.  Note  that  for 
the  classification  purpose,  negative  and  positive  infinities 
do  not  interfere  with  our  process,  although  as  the  values  of 
estimates  they  are  far  from  being  acceptable. 

It  is  frequently  observed,  however,  that  a  researcher  uses 


i 


r 
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a  test  whose  test  information  function  is  bell-shaped,  like  the 
one  for  LIS-U  ,  for  other  purposes  without  giving  much  thought 
to  the  inaccuracy  of  estimation  at  deviated  values  of  9  .  This 
inaccuracy  is  rooted  in  the  test  itself,  but  he  tends  to  blame 
the  method  of  estimation,  complaining  that  they  obtained  positive 
and  negative  infinities  for  some  examinees  as  their  maximum 
likelihood  estimates  of  ability  6  .  When  they  come  across 
this  result,  most  researchers  turn  to  Bayesian  estimation,  as 
if  it  gave  a  solution  to  the  problem.  We  must  note,  however, 
that  the  seemingly  acceptable  finite  estimates  for  the  two 
extreme  response  patterns  are  basically  resultant  from  the  prior 
only,  and  the  test  itself  is  simply  powerless  in  the  entire 
process  of  estimation;  the  fact  which  explains  the  large  differences 
among  the  regressions  of  the  Bayes  modal  estimates  on  ability  9 
with  different  priors,  as  was  observed  in  Chapter  3.  When 
researchers  use  a  Bayesian  estimate  in  such  a  situation,  therefore, 
they  are  simply  covering  up  the  def iciency  of  the  test  they 
chose,  pretending  as  if_  the  test  had  enough  power  to  estimate 
the  ability  accurately,  while  the  truth  is  that  the  amount  of 
test  information  provided  by  the  test  at  those  levels  of  ability 
is  so  small  that  no  real  testing  was  performed  on  these  levels 
of  ability.  It  is  the  researcher  who  must  take  the  blame  for  the 
failure  in  choosing  a  right  test,  not  the  maximum  likelihood 


estimation. 
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As  for  the  second  aspect,  we  must  be  aware  of  the  fact 
that,  as  long  as  there  exists  some  amount  of  error  of  estimation, 
the  frequency  distribution  of  the  resultant  set  of  estimates 
should  be  expected  to  have  larger  variance  than  the  true 
ability  distribution.  It  has  been  pointed  out  (Samejima,  1977c) 
that  for  any  unbiased  estimator,  X  ,  of  ability  9  ,  we  can 
write  for  the  variance  of  X 

(6.1)  Var.(X)  -  Var.(9)  +  E[Var.(X|e)]  >  7ar.(9)  . 

Since  the  maximum  likelihood  estimate  is  asymptotically  unbiased, 

(6.1)  approximates  the  relationship  between  0  and  the  maximum 

likelihood  estimate,  0  ,  when  the  amount  of  test  information 
is  large  enough  for  the  entire  range  of  0  within  which  the 
examinees'  ability  is  located.  If  the  conditional  expectation 
of  X  ,  given  9  ,  is  constant,  i.e.,  Var.(X|0)  =  a 2  , 

then  we  can  rewrite  (6.1)  in  the  form 

(6.2)  Var.(0)  =  Var.(0)  +  a2  . 

When  the  test  information  function  assumes  a  large,  constant  value 

for  the  entire  interval  of  9  within  which  the  examinees'  true 

ability  is  located,  i.e.,  I(9)=C>>0  for  this  interval  of 

6  ,  the  maximum  likelihood  estimate  conditionally  distributes 

-1/2 

approximately  normally,  given  0  ,  with  9  itself  and  C 


as  the  two  parameters.  Thus  we  can  write  in  such  a  situation 
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(6.3)  .  Var. (0)  =  Var. (9)  +  C_1  >  Var.(0)  . 

If  we  use  either  Test  A  or  Test  B  as  our  test,  for  example,  the 
sample  variance  of  the  maximum  likelihood  estimate  is  expected 
to  be  approximately  0.046  larger  than  the  population  variance 
of  ability  9  ,  regardless  of  the  value  of  Var. (9)  . 

It  is  evident,  therefore,  if  some  estimator  of  ability 
6  provides  us  with  an  expected  sample  variance  which  is  the 
same  as  the  population  variance  of  the  true  ability  9  ,  there 
must  be  a  certain  bias  which  makes  the  resultant  estimate  regress 
toward  the  central  tendency  of  the  ability  distribution,  as  we 
have  seen  in  the  Bayes  modal  estimate  in  Chapters  3,  4  and  5. 

We  must  consider,  therefore,  that  the  characteristic  of  the 
maximum  likelihood  estimate  described  in  (2)  is  a  logical  result 
of  the  asymptotic  unbiasedness,  whereas  the  characteristic  of 
Bayesian  estimates  is  a  problem,  which  is  caused  by  its 


biasedness . 
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VII  Estimates  for  the  Two  Extreme  Response  Patterns 

As  we  have  seen  in  the  preceding  chapter,  the  probability 
is  very  low  that  some  of  our  examinees  obtain  one  of  the  two 

extreme  response  patterns,  (0,0,..., 0)  and  (m^.n^ . mn)  , 

if  we  choose  a  right  test.  When  the  test  items  follow  one  of 
the  models  which  satisfy  the  unique  maximum  condition  (Samejima, 
1969,  1972),  like  the  normal  ogive  and  logistic  models,  the 
amount  of  test  information  for  the  range  of  0  in  which  the 
examinees'  ability  is  located  is  a  useful  measure  for  the 
appropriateness  of  the  test.  If  our  test  is  informative 
enough  for  the  entire  range  of  ability  0  of  our  interest, 
as  are  Tests  A  and  B  in  the  examples  in  Chapter  4,  chances  are 
very  slim  that  some  of  our  examinees  obtain  one  of  the  extreme 
response  patterns  and,  consequently,  negative  or  positive 
infinity  for  their  maximum  likelihood  estimates. 

We  must  note,  however,  that  even  with  such  tests  as 

Tests  A  and  B  and  with  groups  of  examinees  whose  ability 

distributes  within  the  interval  of  0  for  which  the  tests  are 

informative,  it  can  still  happen,  though  very  rarely,  that  some 

examinees  obtain  negative  or  positive  infinity  as  their  maximum 

likelihood  estimates  of  ability.  Our  question  is,  therefore, 

/ 

if  there  is  any  way  to  avoid  such  a  situation,  without  losing 
the  perspective  of  objective  testing,  which  we  shall  not  be  able 
to  accomplish  by  turning  to  Bayesian  estimation. 
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From  the  purpose  of  objective  testing,  it  is  obvious 
that  we  should  find  a  solution  for  this  problem  without  using 
any  information  which  the  test  itself  does  not  provide.  In 
so  doing,  we  shall  make  a  population-free  estimation,  in  which 
examinees  are  solely  evaluated  from  their  performances  in  the 
testing. 

Hereafter,  we  shall  denote  the  two  extreme  response 

patterns,  (0,0,...,0)'  and  (m^,m0 . . . , m  ) 1  ,  by  V-min  and 

V- max  ,  respectively.  We  notice  that  the  operating 

characteristic  Pv-min^^  strictly  decreases  in  9  ,  and 

Pv_max(9)  strictly  increases  in  0  ,  as  long  as  our  test 

items  follow  a  model,  or  models,  like  the  normal  ogive  and 

logistic  models.  Thus  we  can  conceive  of  a  critical  point, 

0  ,  which  satisfies 

c 


r?v  (e)  =  o 

(7.1)  J  V"min 

for  9  >  0 

1 

]  P,,  (9)  =  0 

V-max 

for  0  <  9 
“  1 

Let  Q  be  the  product  of  the  two  operating  characteristics 
of  the  response  patterns,  V-min  and  V-max  ,  such  that 

(7.2)  Q  =  P,.  ,  (9)  P,.  (0)  . 

V-min  V-max 

We  define  0^  as  the  point  of  0  at  which  this  product  is 
minimal.  By  virtue  of  the  assumption  of  local  independence 
(Lord  and  Novick,  1968),  6^  is  the  solution  for  the  equation: 
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(7.3) 


¥logQ’¥  108  PV-min(9)  +  ~k  108  PV-max(6) 


n 


¥F  108  px  (9:v0) 

g=i  g  6 


+  i  -W  108  P*  (9lVV 

g=l  g 

n  n 

£  A  (6;x  =0)  +  E  A  (9;x_=m^) 
g=l  g  5  g=l  Xg 


0  , 


where  A  (9)  is  the  basic  function  (Samejima,  1969,  1972)  of 
Xg 

the  item  response  x  .  It  is  interesting  to  note  that  this 

S 

critical  value  6  is  the  maximum  likelihood  estimate  of 
c 

ability  9  for  the  response  pattern,  (0,0, ... ,0,1,1, ... ,1)  ,  on 

the  test  of  2n  binary  items,  the  first  n  items  of  which 

have  P  (9;x  -1)  (g*l,2, . . . ,n)  ,  and  the  second  n  items  of 
Xg  8 

which  have  P  (0;x  *m  )  (g“l,2, . . . ,n)  ,  as  their  respective 

Xg  8  8 

item  characteristic  functions. 

We  shall  aim  at  finding  finite  substitutes  for  the  two 

maximum  likelihood  estimates,  9,.  .  and  9„  ,  which 

V-min  V-max 

are  negative  and  positive  Infinities,  respectively,  in  such 

a  way  that  the  substitution  should  provide  us  with  a  regression 

which  is  close  enough  to  9  ,  i.e.,  the  unbiasedness  of  the 

estimator,  for  some  range  of  9  .  Let  9*  .  and  6* 

V-min  V-max 

denote  such  estimates,  and  9*  be  the  resultant  estimator, 


such  that 
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f  =  Q* 

V-min 

for 

V-min 

(7.4) 

0*  ■< 

0V-max 

for 

V-max 

l  =  §V 

for 

all  the  other  response 

patterns. 


We  can  write  for  the  regression  of  6*  on  ability  0  such  that 

(7.5)  E(0*  1 0)  -  Z  6V  Pv(8)  +  6*_m.n  Pv_min(e) 

V^V-min 

V^V-max 

+  6V-max  PV-raax('6') 


I  0V  Pv(0)  +  9*-min  Vmin(9) 
V^V-min 
V^V-max 

for  0  <  9 


*  »v  V9)  +  6V-max  Vmax(6> 

V^V-min 
V^V-max 


for  0  >  0. 


If  this  estimator,  0*  ,  provides  us  with  an  approximate 
unbiasedness  for  a  certain  range  of  0  ,  (®»  ®)  >  then  we 

shall  be  able  to  write 


(7.6) 


'  Z 
V^V-min 
V^V-max 


Pv(0) 


®V-min  PV-min^ 


for  0  <  0  <  0 

-  “  c 


i 

V^V-min 

V^V-max 


eV  V9) 


6V-max  PV-max^ 


for 


9 


e  <  0  <  0  . 

C 
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In  practice,  we  must  search  the  interval  of  6  ,  (6,  0)  ,  for 

which  such  an  estimator,  0*  ,  is  availiable,  in  relation  with 
a  specific  test  of  our  interest.  From  (7.6),  we  can  further 
write 


(7.7) 


I  0^ 

vyv-min 

V^ V-max 


A 


V6>  *  ev-mln 


pv-»i„(e>  de 


< 


e  e, 

V^V-min 
V^V-max  ^ 


*6 


*  \  (e2  -  e2) 

Z  C 


P,.(6)  d0  +  0* 

V'  V-max 


=  j  (02  -  e|) 


p„  (e)  de 

V-max 


Thus  the  two  estimates,  6*  .  and  0*  ,  can  be  obtained  by 

V-min  V-max 


r°v  m-in  =  ~  02)  "  Z  K 

V-min  L  c  -  ...  .  V 

V^ V-min 

V^  V-max 


(7.8) 


0* 


[^(e2  -  e2)  -  z  e, 


^  V-max  2 


V^V-min 

V^V-max 


I/  Pp< 
J/  Vl 

V  f  »c  Pv< 


(e)  de] 


(e)  de] 


with  some  appropriate  values  for  6  and  0  . 


For  the  purpose  of  illustration,  we  use  LIS-U  again,  and 
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put  our  effort  upon  finding  a  suitable  interval,  (8,  8)  ,  and 
the  corresponding  two  estimates,  0*  .  and  0*  ,  which 

substitute  for  negative  and  positive  infinities,  respectively, 
in  the  maximum  likelihood  estimation.  We  must  be  aware  that 
this  short  test,  LIS-U,  is  not  designed  for  estimating  a  wide 
range  of  ability  8  with  high  accuracy,  as  was  explained  in  the 
preceding  chapter.  This  implies  that  we  should  expect  the 
interval  (6,  6)  to  be  a  relatively  small  one,  for  which  the  test 
information  function,  1(6)  ,  assumes  reasonably  high  values  (cf. 
Figure  3-1.) 

We  obtained  -0.00880  for  the  critical  value,  9C  ,  for 

LIS-U,  which  is  the  solution  for  (7.3).  For  the  endpoints  of 

the  interval,  9  and  0  ,  we  used  eleven  different  sets,  ±1.50  , 

±1.75  ,  ±2.00  ,  ±2.25  ,  ±2.50  ,  ±3.00  ,  ±3.50  ,  ±4.00  ,  ±4.50  , 

±5.00  and  ±5.50  ,  for  the  purpose  of  experimentation.  The 

resultant  set  of  estimates,  6*  .  and  0*  ,  which  was 

obtained  by  using  each  of  these  eleven  intervals,  is  given  in 

Table  7-1.  We  can  see  that  the  value  of  0*  .  decreases  as 

V-min 

the  lower  endpoint  of  the  interval  6  ,  decreases,  and  that  of 
9*  max  Inct63565  as  the  upper  endpoint,  0  ,  increases,  as 
is  expected  from  (7.8).  In  order  to  find  out  if  (7.1)  is 


TABLE  7-1 

Eleven  Sets  of  Estimates,  0*  ,  and  9* 

V-min  V-max 

of  Ability  for  the  Two  Extreme  Response  Patterns 
(0,0,..., 0)  and  (1,1,..., 1)  ,  Obtained  on  LIS-U 
Using  Eleven  Different  Intervals  for  (0,9)  . 


!,  e 

9* 

V-min 

9* 

V-max 

±  1.50 

-1.47883 

1.52237 

±  1.75 

1.64702 

1.65605 

±  2.00 

-1.79255 

1.77649 

±  2.25 

-1.92540 

1.89233 

±  2.50 

-2.05136 

2.00754 

±  3.00 

-2.29490 

2.24127 

±  3.50 

-2.53641 

2.48011 

±4.00 

-2.77945 

2.72254 

±  4.50 

-3.02430 

2.96720 

±  5.00 

-3.27051 

3.21329 

±  5.50 

-3.51765 

3.46032 

-90- 


VII-8 


were  computed,  and  presented  in  Table  7-2.  We  can  see  that  these 
values  are  very  close  to  the  areas,  which  are  obtainable  if  we 
include  all  the  response  patterns,  and  are  equal  to  (9  -  9)  and 
(9  -  9  )  ,  respectively;  in  fact,  the  discrepancies  of  ST  and 

C  ia 

Sy  from  these  values  are  approximately  0.00037  and  0.00035  , 
respectively,  in  each  of  the  eleven  cases,  the  fact  which 
indicates  the  satisfaction  of  (7.1). 

The  regression  of  6*  on  ability  6  ,  which  is  given  in 

the  first  two  lines  of  (7.5),  was  computed  by  using  each  of  the 

eleven  sets  of  6,*,  .  and  6*  ,  and  the  first  five  cases 

V-min  V-max 

are  presented  as  Figure  7-1.  In  each  of  these  five  graphs,  the 
regression,  E ( 6* | 9)  ,  is  drawn  by  a  solid  curve,  and,  for  the 
sake  of  comparison,  the  regression  of  the  Bayes  modal  estimate  with 
n(0,l)  as  the  prior  is  plotted  by  dots,  together  with  the 
solid  straight  line  which  indicates  the  unbiasedness.  We  can 
see  in  these  results  that  all  the  regressions  of  9*  on  9 
are  closer  to  the  unbiasedness  than  the  regression  of  the  Bayes 
modal  estimate,  and,  in  fact,  for  the  first  three  cases  in  which 
the  interval,  (9,  9)  ,  is  (-1.50,1.50)  ,  (-1.75,1.75)  and 
(-2.00,2.00)  ,  respectively,  E(6*|0)  is  very  close  to  the 
straight  line  within  the  respective  intervals,  (9,  0)  .  The 
departure  of  E(9*|o)  from  the  unbiasedness  becomes  greater  as 
we  change  the  interval  to  (-2.25,2.25)  and  (-2.50,2.50)  ,  the 
result  which  was  anticipated  from  the  test  information  function 


TABLE  7-2 


Sum  of  the  Areas,  ,  Under  the  Curves  of  Pv(6) 

Excluding  P  (6)  ,  for  the  Interval  (0,  9  )  , 
v-max  —  c 

and  the  Sum  of  the  Areas,  ,  Under  Py(9)  > 

Excluding  ^(0)  »  for  the  Interval,  (9^,  0)  , 

Together  With  Their  Sum. 


Efl 

SL 

su 

Total 

±  1.50 

1.49083 

1.50845 

2.99928 

±  1.75 

1.74083 

1.75845 

3.49928 

±  2.00 

1.99083 

2.00845 

3.99928 

±2.25 

2.24083 

2.25845 

4.49928 

±2.50 

2.49083 

2.50845 

4.99928 

±  3.00 

2.99083 

3.00845 

5.99928 

±  3.50 

3.49083 

3.50845 

6.99928 

±  4.00 

3.99083 

4.00845 

7.99928 

±  4.50 

4.49083 

4.50845 

8.99928 

±5.00 

4.99083 

5.00845 

9.99928 

±  5.50 

5.49083 

5.50845 

10.99928 

Unclassified 
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FIGURE  7-1 

Regression  of  the  Maximum  Likelihood  Estimate  vlth  Those  for  the 
Two  Extreme  Response  Patterns,  (0,0,..., 0)  and  (1,1,..., 1)  , 
Replaced  by  ev-min  an<*  ®V-max  *  ResPective^y*  on  Ability 

0  (Solid  Curve),  Together  with  the  Regression  of  the  Bayes 
Modal  Estimate  with  n(0,l)  as  the  Prior  (Dotted  Curve)  . 
These  Two  Estimates  Were  Obtained  by  Using  0  -  -1.50  and 

0  -  1.50  . 


REGRESSION 


I 
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0*  .  and  0* 

V-nln  V-max 


FIGURE  7-1  (Continued) 


Were  Obtained  by  Using  6  ■  -1.75  and  $  -  1.75  . 


REGRESSION 


-9 


FIGURE  7-1  (Continued) 

8*  ^  and  0*  Were  Obtained  by  Using  8  ■  -2.00  and  0  ■  2.00 

V-min  V-max 


REGRESSION 
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-5.0  -4.0  -3.0  -2.0  -to  QO  1.0  2.0  aO  4.0  5.0 


LATENT  TRAIT  0 

FIGURE  7-1  (Continued) 

0V-nin  and  ®V-max  Were  Stained  by  Using  6  -  -2.50  and  6-2.50 
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of  LIS-U  shown  In  Figure  3-1.  As  we  can  see  in  this  figure,  the 
test  information  function,  1(6)  ,  assumes  very  small  values 
outside  of  the  interval,  (-2.00,2.00)  ,  which  are,  in  fact,  less 
than  unity,  and,  therefore,  we  should  net  expect  it  to  measure  the 
individual's  ability  accurately  outside  this  interval.  Thus  it 
will  be  best  to  use  -1.79255  and  1.77649  as  the  substitute 
for  the  maximum  likelihood  estimates  for  the  two  extreme  response 
patterns  for  this  test  with  the  above  restriction  of  the  range  of 
0  ,  or  to  use  either  one  of  the  two  sets,  -1.64702  and  1.65605 
with  the  restricted  range  of  (-1.75,1.75)  and  -1.47883  and 
1.52237  with  the  restricted  range  of  (-1.50,1.50)  .  These 
are  suitable  selections,  considering  the  fact  that  the  least 
value  of  the  maximum  likelihood  estimates  for  the  remaining  126 
response  patterns  on  LIS-U  is  -1.3167  for  the  response  pattern, 
(0,0, 0,1, 0,0,0)  ,  and  the  greatest  value  is  1.3028  for  the 
response  pattern,  (1,1, 1,0, 1,1,1)  .  Similar  graphs  for  the 
other  six  cases  are  presented  in  Appendix  as  Figure  A-l.  We 
can  see  that,  as  the  interval,  (6,  6)  ,  becomes  larger,  the 
departure  of  E(0*|e)  from  the  unbiasedness  becomes  greater, 
which  indicates  that  these  sets  of  estimates  are  less  and  less 
suitable  for  use  as  0^^^  an<*  ev-max  * 

The  maximum  likelihood  estimates  for  the  other  126 
response  patterns  on  LIS-U  are  also  presented  in  Appendix,  as 


Table  A-5. 


-98- 


VIII-1 


VIII  Discussion  and  Conclusions 

Bayesian  estimation  was  considered  In  comparison  with 
the  maximum  likelihood  estimation  from  the  standpoint  of  the 
objectivity  of  testing,  which  Is  closely  related  with  the 
unbiasedness  of  estimation  and  the  population-free  estimation. 
Using  several  different  types  of  tests,  including  both  paper- 
and-pencll  tests  and  computerized  adaptive  tests,  the  effect 
of  priors  on  the  resultant  estimate  was  observed.  It  was 
pointed  out  that  the  use  of  priors  in  Bayesian  estimation  will 
result  in  biases  which  favor  certain  individuals  over  certain 
other  individuals,  even  though  they  are  exactly  equal  with 
respect  to  their  ability  levels.  An  alternative  method  of  using 
the  maximum  likelihood  estimation  with  the  replacement  of 
positive  and  negative  infinities  for  the  two  extreme  response 
patterns,  (0,0,...,0)  and  (m^.mj, . . . ,mn)  ,  by  a  pair  of  new 

estimates,  ®v-min  an<*  9V-max  *  was  ProP°8e<*»  an<*  the  resulting 
regression  shows  less  amount  of  bias  than  Bayes  modal  estimate 
does.  We  must  emphasize  that,  unlike  Bayesian  estimates,  this 
modified  maximum  likelihood  estimate,  9*  ,  is  a  population-free 
estimate,  so  that  all  the  individuals  on  the  same  ability  level 
are  treated  fairly  and  equally,  regardless  of  the  populations 
from  which  they  are  selected. 

There  exists  some  philosophical  problem  in  Bayesian 
estimation  which  we  must  call  our  attention  to.  In  any 
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Bayesian  estimation,  we  assume  the  interchangeability  of  individuals 
who  are  assigned  to  the  same  prior.  The  idea  of  measuring 
individuals'  ability  itself  preassumes,  however,  the  heterogeneity 
of  Individuals,  even  though  they  belong  to  the  same  population, 
which  implies  that  the  individuals  are  not  interchangeable ♦  In 
addition  to  this  fact,  it  should  be  noted  that  the  assignment 
of  an  individual  to  a  specific  population  is  more  or  less  arbitrary. 
Most  researchers  use  such  attributes  as  age,  sex,  ethnic  background, 
and  so  forth,  for  defining  populations.  Note  that  they  are  only 
a  partial  information  about  an  individual,  even  if  we  combine  a 
few  of  these  attributes.  Thus  it  happens  frequently  that  a  Ph.  D. 
in  psychology  with  a  certain  ethnic  minority  background  is 
assigned  to  the  group  of  Ph.  D.'s  in  psychology,  or  to  the 
group  of  people  with  the  same  ethnic  background.  The  resultant 
two  Bayesian  estimates  for  this  person  can  be  substantially 
different  from  each  other,  depending  upon  the  difference 
between  the  two  priors.  To  avoid  this  contradiction,  we  must 
accurately  specify  the  population  to  which  each  individual  belongs, 
taking  the  intersection  of  thousands  of  factors,  including 
sex,  age,  education,  ethnic  background,  etc.  If  we  do  this, 
we  will  end  up  with  assigning  each  individual  to  his  own  prior, 
which  is  shared  by  no  one  else.  If  we  know  such  a  prior,  however, 
we  do  not  need  to  test  him  at  all! 

From  all  aspects,  we  must  conclude  that  the  common  belief 
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In  the  superiority  of  the  Bayesian  estimation  over  the  maximum 
likelihood  estimation  in  the  ability  measurement  is  a  farce, 
and  the  additional  information,  the  prior,  is  nothing  but  a 
resource  for  the  biases,  which  may  lead  to  unfair  personnel 
selection  and  other  serious  social  issues. 
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TABLE  A-l 

Sequential  Results  of  Simulated  Tailored  Testing  for 
Hypothetical  Examinees  of  Eleven  Different  Ability 
Levels.  Maximum  Likelihood  Estimation  (MLE)  Was 
Used  for  Ability  Estimation. 
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•>!  ’.V  ••  5 1 8 . 1 3 1 

(■  20.395: 


I 
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TABLE  A-l 


(Continued) 


9  -  -1.75 


Item  Score  MLE  _  _ _ 

rJSJtsL  _ - 

2  _ _ 2 _ _-_L=0.#«3«. _ — : - 

_  3  ■  • _ j_  _ _ t  -i, am _ s_: _ i.. _ e.*oe_ 

&’  5  !■<.;  •  i ■- tt’ZVnUuK --.j, .  ■-•-i.95ia^;V  4  £  >  3*500x' 

i '■:■■•.•  z  ;«?A: i y‘li>  ^iv.  J-iillAUgax; 

J  '  ->  _  1  1)4K^  ' 


Number  of 
Iterations 


Information 


_  2  _____ _ ri.#4«>i»_ 

j _ rl'.9J24.,_ 

l  ,  •  ,.  -2.PQ19- 


.Vasal. 

*'*29.1 

4»064„ 


6  -  -1.25 


Number  of 
Iterations 


Information 


»7.243_ 

A?44A_ 


”T7" 


m 


-106- 


TABLE  A-l  (Continued) 


9  -  -0.75 


1 

i 

‘.4.  _ 

_ .4  _ 

5  •  ‘ 
",  a  ' 

/•7 

u  _ 

9 

_ 10 

if-,  - 
■:  Ur., 

-Li  3  ul. 


Item 

■i  i...) 

9 

7 

_  .  5_: _ 

•-  5  . 

5  i  . 

iL.S — -i 

3  . 

4  , 

_J _ , 

f  j  ■  ' 

V  1 :  3  •. 

_ 3 


Score 


Number  of 
Iterations 


Information 


-  1  ,  .  0,9407 

I  _ _ 0.4011  _ 

_ 1 _ _ -ft,04  75 _ 

1  -0.2520. 

1  ■  -0.3811 

■  l  V  •  -0.4714  l 

...  .  1  _  -0.  7827.. 

'•  . 2_  _  -0.6865  : 

i  2  ~  -0.6208 _ - 

•  2  V  -0.5709  7 

!  K.2  »V-,;  :>’•»•’  .  -0.5307  V 

_U.i _ _ _ tQ,  6,469 _ 


5  _ 

_ 4.1-.. _ : _ _ _ J.U9._ 

5  _  4.695  . 

.4  ,  :v;,^/.6.053 

=»-.  4  >  7.144 

.  . 4  .  V  g.  04J»  i 

__  4  .  ...  17.260 

4  : _ _  _  __  ._10.269_ 

3 _ _ _ _ _ »2.720_ 

.  .3  .,T„  14.810 

..3  .>vV 16. 642  : 

.3.  '  :  _ _2 1,053 


0  -  -0.25 


.  2 

...i .... 

'4' _ 

9  • 

•  .  6'. 

^  7.  • 

a 

10 

i(  :'" 

M2  :  . 

_iAl, 


.  3 

_  J  _ 

•••  5  : 

■•••  3' 


..‘Is# 


Score 

j'iii 


>  1 

2  •  •- . 

■  .  2  ’ 
„..L\ . 
,2 

_ 

t  w4;4  a  •  J 

tei  ! 1 ,  w; 


Number  of 
.E  .  Iterations 


Information 


-0.8834 _ ^  .5 _ 

-0.4710,  .  ___.4 _ 

.-9 .  J430___, . _ _4 _ „ 

-0.48  70  4 

-0.4081  /  .j'.’J 
— 0»l4o8  ,  •  *4 

-0.2515  _  3„. 

-0.0905  _ 4^ _ 1 

-0 . 17 94  3 

-0*2471 . .  iTlJ’.. 

-0*3015  ,  3  :/-a 

-0.3469-  •  ,r~3 


1 _  3,503  _ 

_ _ 4,  6  3  5 _ 

'. !  .  6.796 

8.111  *, 
:i.ui  ^:|.579'_ 
L_  11.055. 

_ 12.524 

i  14.751 

,  *  'j  i.  ’.I  16*  79A  « 

.'l-MV-18.62  7  ■: 

.'HL^SjuuuhL. 
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TABLE  A-l  (Continued) 


0  -  0.00 


Score  MLE 

._.2_r_ ..  j _ rfttWM 

.2. 


Number  of 
Iterations 


Information 


-•2  .  V.  i  0.04VJ 

t.i  W.0.25JS 


.5.  _ 

r0..47iU_:_Lj»_!. 
-0.3430 ~  4 


j:  •*?**  '  •*  i.'  *.  •' 


_1.503_ 

4*633_ 
076  > 
172  • 


.0,1*36 , _ ii  4- _ LI  10,664_ 

»  •*  .  '  ~  n  d.  I  l  ■  aj,A 


L 

.X 


.0,2464 


7’1~  1 ,-.•  .. o. o  142 \i\ m y. i. ;^T ' -„. 41,; .r.n* 

•.s  2  j . a u4:-4^v.-o,o»?o-f* .-3 .'a'^v'-y.:  i-^wi 

i ...  >.V: 


111,96*. 
__ii,0Ql 
T^T£l7*5«3~ 
"  241'  ■ 

*i,jjuu 


9  -  0.50 


Number  of 


Item  Score  MLE 

*  .2  _  .  _9.  .  u-l  _.0,9407__ 

7  J ^-..IJ _ 0,4011 

J4__  L^P^JZ Z  <  „  ■■„  Pt.J,  9,  6j95 .  ...4 


Iterations  Information 

^  •;  :  ■•  4- •  vf 


3,119.. 

'  ,3-07- 


•  s  .i:;v...7v7^v  i - v  / ;  v  -;  v<>  o . 4 u»i»  i/v 3  6.060  , 

J  »«3>2I 


0. 7390 


„? _ :_ 


'.,  1  '  L.-1-1  ' _ 0.6370 _ •■»  J 

1  '  1  0.3V13  3  • 

jj‘  1.^-  0,5420.x  >%, 

,496? * 


10,365_ 

12,067- 

11*544, 


■x"<\.r..3  -JZit”'.'  14. 65 l’j 

16.026,; 

!&2  &’'*/**  13.  739  3 


19,795; 
UJ1,734 
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TABLE  A-l  (Continued) 


0  -  1.00 


Score 


Number  of 
MLE  Iterations 


Information 


_ l.o.V407_ _ S.._ _ _ : _ _ 

_1,5104 _ 4 _ _ _ 2.664_ 

„  _  1.1 662  _  .4 _ : _ _  4.921 

2  •  i  : '  ;  ••  1.5034  .  .  t  3  ...  •  r.J,  7.267 


- l.39U__  ...3 

1  _ _'_i*3P*.9,j _ •  .  3 

2  1.4186  3 


'  '.Vi'- •  I.- 
’•  •»  ■■ 


9.476 

111,245.4 


_ 14,089  _ 

_ 15.989 


tlT^7v  8,:  ■>$  3  -Vv  5-  V  Af  bfc  0 . 6>  a_ . 


0  -  1.50 


Number  of 


Item 

Score 

MLE  Iterations 

Information 

■  i: 

,.„T  ■ 

,2  .j. 

*  ’  •  ‘  *  • ,  ’  .  *#' 

i 

9  .  _ 

..  l 

■.  ,u  0.940  f  .‘5 _  .. 

• 

3 

7 

.  2  _ 

'  ;  1,5104....  .  4  .  _ 

_  2.664  . 

4 

l 

rr-  -  1^1*“  ...  .  4  4-,^- 

.  _4 ,92 1  __ 

..  5 

8  . 

t  .:  - 

;  .1.0231  .  .  4  .!•  •' 

.  6.407 

V.  * 

•  ’  -  •  .\  v * 

2  .  •• 

1.1953:.-  ..  3 

•  !•  753- 

l*  f 

_y  8  .  V 

_ ; 

-..v,  ,  i.41^2  'W...-* 3 

^lLii»iir  • 

8 

8  ' 

2  ... 

_  l.j»485 _ :.  3 _ J _ 1 

13.121 

.  « 

.  :  8 

..  1  __ 

.  1,4398  '  3 _ _ 

15,665 

10 

8 

1 

1.3593  .  _  .3  ,  „ 

_ ,11.141. 

Tu 

...  8 

1 

i,  :V.  '  ,i .29by, ..  3  •?.!•.♦  \i" 

19.757 

.12 

•  t  .  v  5  ► 

*i‘*  1 
:*  ;• •  *. 

\  -  1.2438  ? 1 .3  .  ..y'7i;> 

A:*  21^414  , 

> . .  «  ?  * 
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TABLE  A-l  (Continued) 


0  -  2.00 


Item  Score 

■'  i  ill  :J 

,2 


Number  of 
HLE  Iterations 

•  ,* ;  %<•  1  ■  '*  iff  r». 


9  i _ 

i _ 9.  .11:  — l _ ._L _ 1* — ;.. 


Information 

*•  I  it 

. .  *  •  ' 


’  5  ~ 

*9 

4'.r?.r77  "-Trr’ 

T~*"rr 

6 

.9  . 

*.«  1 -t  .‘:  2.501  i-/? 

s7:j.' 

.•'A 

t 

*.»  ill. 

ill  .  2#2S7jLll. 

4-;.. 

v,!*.  s 

a  ...  . 

9  .. 

1  _  .  ^  2. 1201  . 

.  3 

9 

9  . 

9 

.  .1;. .2^0001  .... 

..3 

10  . 

1  r  ■  '  1.901/  ,. 

3 

_ 13 

11 

8,. 

:*•.  2‘  4,...  r  .1. 8583.0  .5 

3.-: 

<  i  «!'-l  * 

12  \  ,.,'j 

81..'  . 

.S  !«•«’ ..  •  I  •  14  72  74  f  \ 

,W3  , 

■■*  £  4^V^17 

‘.Miiii.  1 

a  1 

=--:.2*.  ■  1.7 7 83 

V-  ? 

sA*ywcu 

14  (l 

a  ... 

•—’.2s.  •.._  ..,.  1.8{Ug _ 

,.3 

21 

■.'•I 


.913 


0  -  2.50 


.ii 

i  . 
i 
4. 
.  5 
6 

111- 


Item  Score 

: ... .:.  5  .  ■- ...  2.  _ _ 1-1. 

v 

9 . 
a 
v 

9 


Number  of 
MLE  Iterations  Information 


m\7 


l 

'iZlll 


_. x . 


_ 2,0904  _. 

.  _  2.1*19 

,t2  .  -iw,  ” ■  2.3430  ‘r- 

9  ■  •  2 _ _ 


4 

_4; 

4  :.r  1''  ■ 


9  2 _ _ 


•,  2.47ttSi.- 

-•t  4-;. 

^2.5784 

2.6563 

5 

'  * 

2.7197 _ 

4  ' 

'  >•  •  6 

j£\4. 751  v 

,v 

.1  • 


.325 


l*.LU_a 

'81  #3^ 
85  79 
89^0 
2.9244 
_ 2,9327 


7.1 33, 


...  :•» va*H40 

'  ;,9  n  -  .?2; 5966 

icl  1'ljiktJ..* lltx. 

_  .9 _ •.v._.2.__:^ _ 2,05.99. 

......  9_ _ _ 2.5415 


JZ71F* 7.485,1 

S^JLLaaB!a2figJaJt.0IJ 

_ _ _ :  J.388... 

I.648_ 

, _ 9,.m_ 

_. .'  4  *  7^13. 715 

**:  4..i'^/:a  ;/.V"A'.r  13.441  : 

_ 18,467  . 

J.LUIli_ 


! _ 4-1. 

_ il 


1* 
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TABLE  A-2 

Sequential  Results  of  Simulated  Tailored  Testing  for 
Hypothetical  Examinees  of  Eleven  Different  Ability 
Levels.  Bayes  Modal  Estimation  (BME)  Was  Used  for 
Ability  Estimation,  with  the  Prior,  n (0.0, 1.0)  . 


0  -  -2.25 


Humber  of 


Item  Score 

BME 

Iterations 

Information 

•71/v'.’"**  *--7.vT 

i.  *.\} .  ( 1:7  V* 

7-  0.0"’"; 

•.:.'v.-0t53#5i1' 

17'--l*i?8'L'_ 

V  /  ,  »’•*  ’  •  *  T 

.  -p  *-  *  ,#-ii  •'**  *  Vl 

■  '  ‘!s  jV  l.454»: 

.,7  5  •••'•A  2.  726  il 

:  3  .1 

-1.4223 

_ L.V-_ _  -uj 

4.053. 

5.—  ..  ■  3_  1  1.  _i_. 

.__rlj  5456.1 

4  i 

,4,977 

-1.6282 

4 

5,679 

-*'•  .1 3  2  "  V>>  *'  -  ■ 

r!;t 

,2j» 

*  1  -1  i-7,95  3  5 

tiiv.Vl.9286 

ilir2..W340i: 

'■*X-  4. v-'  Vj  ;vi».6i5*373,'.!: 
(-S*.i4‘,f?S;Vv*  &*fe.5.394'« 

?2  -‘dr :*v  i  ?i'«V 0-2.0303 •  -;:•}>» 4 V  v;0  wn. 

•ajSbatii tufz&sUL?  2.073  2 , 


r.2 


3  __ 

3 .. 

4 


6.185— 
A*A90— 
409  T 
887  i" 
35611 
12.812— 
_13.252  . 
13.4  75t_ 
080  .» 
469  X 
842.x_ 

.  2 _ _  1 1— u t2.2/96 _ _ _  4 15.199 

1  ■■  •  -2.3004 _ _ 3 _ .  15.542  _ 

87l.„ 
809  " 
J94 


_t2,1904_ _ 

— 2.24iiL, 
.9812 
.0303 
.07351 
_r2»H2l— 
-‘2*146  7— 
-2.J/8A- 


:  2.".T*X-“  l  .''-2.2067  0-''30.^/.-,’V .?  >..14. 

I’;'.'*  2,2329'’  •  K.?.3:  *  Vf.* 

.  liilLlimiiri  ..25.7 LV;  ll/j-  3,fil_2l^t^dil4. 


2  •  •  I  “ _ L_.-2.3004  _ _  3 _ _ 15. 

.  2_^_  -  - - -  _*_2 , 3ly.9--._3._- - 1 5 , 

2...  -W.  '•»!,  .,-2l2479..’.  i.><-  5-  3,. 

2 ir-iv^'-?-2.2665  u 3 -■)/.' *r>; ;f <-y;u. 


..—19. 
.20* 


545,;. 

922_ 

248. 


TABLE  A-2  (Continued) 

Number  of 

BKE  Iterations' 

Information 

;rj.AiCZ 


1 _ r  l.S6<:0 

.  ,,‘--i  .4*50 _ _ 

l.Mii'X.ri!* 

%  iii.wA.V''  ..'• 


\3T,:u\  >: a' 


nnm 


•  ■*-*#-  ,_j 

1.454. 
i.lttL 
7*0  . 
42? 
9.01_ 

7l3i 
0491 

?«79<L 
„_15**1?_ 


N7*ii 


f  *V 


1 
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TABLE  A- 2  (Continued) 


8  -  -0.75 


Item  Score 


BME 


Number  of 
Iterations 


Information 


■  ••  .  t*-  *  Tc**  *+' 

,  ,  ili\\  i  j  it  > 

•  •■.~TT\^+  o.o  .  f , 

•i  iv>r:^>>V“0*53a5*5.:?^  5  H  i 

,  .vi'  ►*  / ; 

^  4  .i;*54-,:, 

v^' 3. 150  ;* 

> 

1  -0.5495  3 

*,788 

•  5  3  •  •  , 

.2 _  \ -0,4260.  3 

6. *29 

,  6  .  5 

1  .  -0.51*1  3 

8.087 

7. ..v .3  V.-.j'J'-- 
£■•••  ~v. -V >'•»' 

A-9 

2  v.V.  - o.**99  ..3 

2  * i’  h  ••  •  •  *;.  h -o . *0 1 9  v -3;jv 

l  l  »>iT'l'  i’V>;:'-o.462*  »  L-  . .  3  ■  ::■) 

9.713^7 
J 1 1  f'0*9  /S  j 

.'/Vv>'v^.'.12.99afV. 

2  *  -0.4233  3 

1*.38B  . 

-  - 

U  5 

2  -0.265*  ,  * 

15.069 

12  S _ 

4_>  - :  ifiL32lJi _ _L_S 

_ _ 17,217 

Li- ■  “V.iiU 

U  .  '  .v-jr-o. 2.073  .•.JVfilVSi'.-  4^V^;3'18.56*  : 

A-o«l2$#i  3-^ J».*W 

6  -  -0.25 


It 


Score 


Number  of 
BME  Iterations 


Information 


TABLE  A-2  (Continued) 


9  -  0.00 


Item  Score 


BKE 


Number  of 
Iterations 


Information 


TT—  ’*»  T? 

0e5J85 '  5 

<  »A.-*kro*j»93;,^d  ••  3 

-0.5195_  •  ’3 

.-0,4268 _ U _ 

O.SlHt  3 


lif- 3 


T“ 


V  3  % V‘;,\-0*401 9 

.-/Lr 

1 _ -0*4233 _ _  ’  T 

T  -A  lil  C/. 


-3_: _ : 


„lt.l3<Lj£ 
4.iaa_ 
_ _ t,429_ 

..  a. oa / 

Vi': 9.713-5 
-^  V  v  ."=11.049 

-—14,388.-. 


LAl _ 2 _ '•  '  -o-l 

448  _ _ _4  16.148_ 

tTA 

i 

8.566 

O.pll^; 

J.’.K*.  t'.'J 

0  -  0.50 


Item  Score 


HME 


Number  of 
Iterations 


Information 


•.  1  'Vr  .0.0  '  :  !’  i... ..  .  .  1  ..  , 

•»  i.*4»4 

1  -iiilLli. -'2.. •■■■:.  '■':  .'■'  ‘1^:1  i:l 5 9 /,'  H  5;4 .  -Al,'.  v"  ■'igir.-.V-  i.  1 $4 


4 

.5 

_4 


8  . 
7__ 


.1 

2. 

J. 


—  ;.0*,S034.i...:i.  3 

_ _ 4 

WM.U^.1 


,»vf  ,» ,  «  ; r$v..t  i •  v>;*’  :;•••  * 

-A*. .'-.4..8^.^:.:a.l£i .•a;o69a  »^3 

.10 - 7.-1; - - .2_j_; _ ;• _ 1.1537  _ 3. 


7T 


12  ,  a 

L  •*  *  ' 

ni^.-vT-'V^V 

•,*£;  -.nM 

1.1537  . 
.1.1036. 
J  ..0620. 


15«t_ 
4.0 54. 
5.890, 
.—^,7,764.. 
4 /  9.255  i 
.  ..^rfj.vvf'lli.572? 
;A3^l2^,t3.l53ix 
1—15.392. 
H  I.  034 
^8.4911 

......  ..  ...  ... -19*464 
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TABLE  A- 2  (Continued) 


6  -  1.00 


4 

......  >. 

Tv  7 

_IQ 
,.*i; 
— M 
.  u  i 
»•'  14. 
.L;  4>_ 
-.16 


Number  of 

Score  BME  Iterations 

■  a.,:-;*  .■ .W-  o.o  ■<  7;V.  -••TIT".; 

1  "•.  }*'  “:V  0.5385  r  .£  5  ' , 

JLZ.slJu _  V-  M397,;  li-l»  .  __;.J 

; _ 1  _ 0.9834  -  1  3  .  j_„ 

_  1  _ 0,77bd _ 3  _ 

■  _  1  _ 0.654$ _  3 _ 

/.  1 V  .4 0.3466  ,  “..V  3  7 

".V  I ■: ai.fr  ?/W  ■  0.4<m- V'LvJ  .^V-v;V.' 

2  •<•  •  :<•  o.ssia'iV  t  3 


ilLit  2.  rsr,v/  -1  :  .o.  5*is. 

_  l _ ___  0.4493 

_ 1_  2  '  '  .  "0.5423 

_ _  t  0..494J 

r"Tf  2  '  •;>:  .  .  0.535  7' 

••  l  ■■  0.4993 

__J  2  :  .0.5300 

_ 2 _ _ _ 0,>5«3 


li.0,55i81tV_i_J_'^ 

0.4993  _  3 _ _ 

0.5423_ _ -J _ 

0. 4943  3 

rV 0.5357  .  -i:f 3  ’ 

••  0.4993  -3  '  ?  ' 

„0.J5309_J.U,3 
0.6383  3 _ _ 


_ Information 

•  /  1.454 

LlLi.'_c.2U56_ 

_ _ _ 4.054  ... 

_ _.5.597 _ 

6.  93  <L_ 

II.;//'  8.100  . 

V ».12»’.*S 

t.l  j'X  -I0.931,?J 

_ ; _ 12.05b1 _ 

_ _ 13.800  _ 

_ :J4.942__ 

■  >-•  •  16,690  ..- 

■  w'-  ‘  17.928  •-' 

JL _ :..lit592_^ 

_ _ 2 MU 


e  -  i.5o 


Item  Score 


TTv? 


Number  of 
BME  Iterations 
0  ’  '  .  7.  f1:*  •  /  f 


•*  .  4.  i  !.-*  .4  .v  >  V  ■  '-O.o  •  . 
fV  3  ."i  7-tit  2  iy  0 .  53  8 5  f  '7  >5  '«  7 


y*T 

4*  " 
-  10  ; 

:1L 


J.  »L_l.x..2 _ _• _  1 .6435  .; — 

8  ,  11  1 _ V  J725_ 

_ b__i _ i _ _ _ ^iii68d_ 


Information 

•  Tv 

1.454  <’ 

l_ii.J«1564l  . 

445... 

_ 6-.25.9_ 

„„  •  J..2.46_. 


^r:  2  '  ?-k,  \ VZ  ~  l  • 6  94#  .yjU V*  ■  4-  9.875.'; 

Z+  8  &  tfL-L  •>'  vVi  1 . 3435  .‘ftf  V  ’&$>  C/kv-12. 86O'; 


_  8  _ 1 _ .3489 

.....  8  r*  .  2  •  '  1.4569 


1.4569 _ ,..3. 


TuT.-tVi' -V 


_ -17.495. 

_ 4. 19. 936_ 

.  .4.,;-  .22.120- 

'  k. .*•  r  *  r  » 

Ju*.  «*■< 


116 


TABLE  A- 3 

Sequential  Results  of  Simulated  Tailored  Testing  for 
Hypothetical  Examinees  of  Eleven  Different  Ability 
Levels.  Bayes  Modal  Estimation  (BME)  Was  Used  for 
Ability  Estimation,  with  the  Prior,  n(0.0,0.8)  . 
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Sequential  Results  of  Simulated  Tailored  Testing  for 
Hypothetical  Examinees  of  Eleven  Different  Ability 
Levels.  Bayes  Modal  Estimation  (BME)  Was  Used  for 
Ability  Estimation,  with  the  Prior,  n(0.0,0.5)  . 
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TABLE  A-4  (Continued) 
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TABLE  A-4  (Continued) 
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LATENT  TRAIT  0 


FIGURE  A-l 

Regression  of  the  Maximum  Likelihood  Estimate  with  Those  for  the 
Two  Extreme  Response  Patterns,  (0,0,..., 0)  and  (1,1,..., 1)  , 
Replaced  by  ®v-min  0V-max  ’  re8Pective^y»  on  Ability 

6  (Solid  Curve),  Together  with  the  Regression  of  the  Bayes 
Modal  Estimate  with  n(0,l)  as  the  Prior  (Dotted  Curve)  . 
These  Two  Estimates  Were  Obtained  by  Using  6  ■  -3.00  and 

0  -  3.00  . 
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FIGURE  A-l  (Continued) 
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LATENT  TRAIT  6 

FIGURE  A-l  (Continued) 

0*  .  and  0*  Were  Obtained  by  Using  8  ■  -5.50  and  6  -5.50 

V-min  V-max  - 
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TABLE  A- 5 


Maximum  Likelihood  Estimates  for  the  126  Possible  Response 
Patterns  on  LIS-U,  Excluding  the  Two  Extreme  Response 
Patterns,  in  Which  All  the  Answers  Are  Incorrect,  and 
All  of  Them  Are  Correct,  Respectively. 
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Number  of 
Iterations 
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Note;  In  this  table,  1  is  used  Instead  of  0  for  the  Incorrect  answers, 
and  2  Is  used  Instead  of  1  for  the  correct  answers. 


TABLE  A- 5  (Continued) 


-A3. 

'■.?* ; 
•.'f-  ,45  :■ 

:  v»-  _  •> 

gr-SJ- 

.  >43- 

'‘49 
rw  • 
W5  0-*. 

*51.' 


HS4~ 

I£s££ 

\ff5X  ' 

— An.: 
-lA2_ 


-25. 


Response 

Pattern 


_ ■  2112121 
-  1212121 
2212121 
U-tU212l  -• 
*- 2122121 

- 1.  222212 1 
-'4-11 1 122 1 
Y;  121 112  21.:  r' 
l ? I I22i_i 

.vf  2211221:.;-' 

-^112122X^2. 

—  122122J_ 
2221221 _ _ 


2 112221  - 
->  1212221  ~ 

— 12L2JL222J _ 

-UL22221 _ 


-212222: 


-2112112 

-12121.12 


MLE 


_»Q,  27c  7 


Mggl&Z 

%&P~ZiQ?jr< 

nTn«r74  - 

-^llO-26-La_l 

-r— fll  A2Z2 _ 

— -Jl.  ?An5  •  • 

•.“:'-o;ooi8--i 

^■a..L938‘y. 

— -=XUa2-X5 


ii.64  :- 
y£»5t- 

1.  22222  21  - 
.*■  U1I112- 

- — ZH  1  1  T7  -  ' 

- 0-50311 _ 

V...  1.0301'. 

I  -0.7214  - 

— 62_ 

- L2111 17 

«,n  ^”7*7^ 

i:a8 

- 2211117 

-I  ft  9 

_ LI  3  1  1  1 7 

.  —  n .  44  s  *? 

jr'70--: 

~3n- 

—  J7 

"  •  ■  .  2121112 
1221112 
- - 2221  n  7 

'.  —  0.  1331  . 
;:,-0.0U4-V/ 

n  “in  a  tr 

*  .73 

_ 11171  1? 

— =0-SQ22 _ 

-=-Q— 24Q^ 


Number  of 
Iterations 


■  -0. 104-5 

:  4 

- SLl21^2__ 

-7.-0.2915  1 

_ 4_ 

.•■  4 

-  r  0.02  40 
_  '•0.17-57 

v>. 

.0.5636 

•'..5 

r.o*4 525'  • 

5 

•U-J.r0'- 1923 --  v 

A--d.  ni  Qt  -  ■ 

>;.  4  .. 

5? 


3 

4 


,y_76  r 

^77 

—7H 

2212112- 
1122112 
-i— 2l22LL2__l 

- 12221 L? 

0.2364 
..-0.2562 
n.  no  n  a 

4 

4 ; 

— IS- 

— 4_ 

— 3C_ 

— 31_ 

- Lin  7  t  7 

MaJOltfU . 

,  „,—n.4i  in 

;32' 

'  2111212  -i 

-0.1609 

4^ 

~  33 

.1211212  '  •• 

•  0. 0066 

•  3  : 

:  77  117]-)  ■ 

„  a~zz3& 

^ ^  Tf. 

Sot«:  In  this  teble,  I  is  used  instead  of  0  for  , 
•nd  2  is  u..d  ln8«.d  of  a  for  c°.f  corr" 


1 

1 


-136- 


TABLE  A-5  (Continued) 
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Note;  In  this  table,  1  la  used  Instead  of  0  for  the  Incorrect  answers 
and  2  Is  used  Instead  of  1  for  the  correct  answers. 
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NAS.  Pensacola,  FL  32508 

Dr.  Robert  G.  Snith 

Office  of  Chief  of  Naval  Operations 

OP-987H 

Washington,  DC  20350 
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Alexandria,  VA  22333 

HQ  USAREUE  4  7th  Army 
ODCSOPS 

USAAREUE  Director  of  GED 
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Special  Assistant  for  Marine 
Corps  Matters 
Code  100M 

Office  of  Naval  Research 
800  N.  Quincy  St. 
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1  Military  Assistant  for  Training  and 
Personnel  Technology 
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Civil  Govt 


1  Dr.  Susan  Chi pm an 

Learning  and  Development  *- 
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Alexandria,  VA  22314 

1  Dr.  Vern  W.  Urry 
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1  Dr.  John  B.  Carroll 
Psychometric  Lab 
Univ.  of  No.  Carolina 
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(Riiversity  of  Iowa 
Iowa  City,  1A  52242 

1  Dr.  Meredith  P.  Crawford 

American  Psychological  Association 
1200  17th  Street,  N.W. 
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Groningen 
NETHERLANDS 

1  Dr.  Robert  Linn 

College  of  Education 
University  of  Illinois 
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University  of  South  Carolina 
Columbia,  SC  29208 


-147- 


Non  Govt 


Non  Govt 


1  OR.  PATRICK  SUPPES 
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Redondo  Beach,  CA  90277 
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1  Dr.  Ron  Hambleton 
School  of  Education 
Ikilverslty  of  Massachusetts 
Amherst,  Mass.  01002 
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V8S  5E4  Canada 

1  Dr.  P.  Mengal 

Faculte'  de  Psychologle 
et  des  Sciences  de  1' Education 
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